I''m working on some scripts for an API in a game and i'm trying to plan out the most efficient route for grabbing a very large database dump through JSON.
I would like to download everything in parts and update it at intervals. so say every 15 minutes fetch 1500 users of data and then update them with a chron job in the sql database.
If you have any better ideas i'de love to hear em' This has been somehting i've been working on for a while now and i'm new to projects of this scale.

This post was edited by Glyph89 on Aug 30 2016 08:32pm

what do you have control over? i'm not entirely clear from your description. if data isn't changing every 15 min and you control the source data, you can put triggers in and just send deltas. there are various sync products available as well if you control the source. one example i've used in the past: https://www.symmetricds.org/

if you dont control the source or the api, what options does the api give you?

1500 users doesn't sound like a huge deal of data, but it depends on how much stuff is associated with it. if it's not your api, be careful about timeouts and getting flagged for abuse.

This post was edited by carteblanche on Aug 30 2016 09:27pm

not a whole lot im using a web host. the api im using. using triggers would not work unless they had a sub trigger (i.e. timeuntil) if i understand what you mean by triggers and the sub trigger would be the delta if im interpreting this right. the api only gives me a way to grab data. that data includes specific key things im looking for. ive spoken to the developers of the api and i've set up the proper procedures so i dont get flagged and such. but basically the best option i could think of was syncing data in chunks. i.e. in football players have stats and they change from each game but until that game they dont change. and i dont know if they changed unless i grabbed the data in one of my "chunk updates"

its a fairly small amount of data ide say about 1-2kb per user depending what kind of call it is. So about 1.5-3MB of data per call. i plan on also storing this in some mysql tables and only updateding the tables if the data has changed for specific users etc... i have that figured out.

The only thing im worried about is updating it more often. if there are say 15k users and im only doing 1.5k users every 15 minutes that's kind of an issue. one update every 2 and a half hours (for a full cycle)

I can grab specific users data but i can't afford sending that many requests it will not work that way. so my only other idea is doing it by chunks.

This post was edited by Glyph89 on Sep 2 2016 08:33pm

the trigger was if you controlled the source data. if you dont control it, then you really dont have many options. just poll as often as you can and do a delta with your local copy as you mentioned.

the only thing i can think of is do you have an api that gets you the new games? then you can fetch users for games that were played instead of all users

edit

This post was edited by Glyph89 on Sep 4 2016 12:57am

why are you set on making it timer?

This post was edited by Ideophobe on Sep 5 2016 01:10pm

Not necessarily set on making it a timer it was the best thing i could think of.
I think just using a button to update it whenever people want works just fine if there were some restrictions on it. an "update" button under each users profile that people could click to update their stats.

This post was edited by Glyph89 on Sep 5 2016 07:20pm

I think the timer makes more sense tbh. 1