I reviewed the implementation and made a few observations. Something that stood out immediately is the time it takes to update an object when the JSON payload (the
castMovies column) is included is about 2.5 times slower vs when JSON is not there. Here are the time for comparison:
There are two highlighted requests here: the first one (1.18 seconds) is when an object is updated and the payload includes JSON. The other one (410 milliseconds) is when an object is updated without JSON column.
This made me think that the current approach is not very efficient. What you’re doing is passing that rather large JSON array back in forth between the Cloud Code and database for the purpose of processing it in the cloud code. And the purpose of that is to fetch data from IMDB for each element in that JSON array.
I thought about what would be a better solution that would remove some complexity. And by complexity, I mean all the recursive calls you have to do. Here’s how I would go about implementing it:
Rather than storing the JSON list as a column in the record, I’d process the JSON (wherever it comes from) and use the
Bulk Upsert block to create individual “movie” objects in Backendless. Basically, every object in JSON would end up being a record in the Backendless database. When you process the JSON list in a loop, use the
Bulk Upsert block for every 100 objects in the source list. Every object would also need the
processed boolean column with the default value set to
Now, the movie objects from the original JSON are in the database, the question is how to organize getting data from IMDB. To do this, I’d create a timer in Backendless (you can do it in Codeless) and set up a schedule for it. The timer wakes up and fetches 50 movie objects where
processed = false. For each object you fetch data from IMDB and do what you’re doing now. As soon as the object is processed, meaning you got data from IMDB, you update that object in the database and set
true. The timer will handle as many objects as it can in a single cycle, however, the same timer will wake up again based on the schedule you came up with and will continue processing the objects. Btw, if the timer gets an empty collection, it means there are no un-processed objects, so the timer would just quit.
Hope this helps.