Hello
My issue is that the case I have is considered a typical case yet it can not complete in the alloted time when I use a timer. A lot of the time is eaten by backendless.
Basicslly I have 2 tables, products and shelf. A shelf can have one product so it is 1:1.
The product table has around 28 attributes (all strings and double, nothing crazy and all are just simply lables). The shelf one has 4 attributes also double and string.
Now what i need to do is to get the shelf snd product data from external source and add them (or update info if they exist ) in the database. This has to be done once or twice a day. Simple ? Good
Alright here is what I am doing
read data from external source (rest call). Takes around 300msec . Thats on me
I get usuallt around 1000 entries in one call.
check if every shelf id exists in data base for every entry to get objectId. I achive it using find-where operation of backendless
-check if every product id exists in data base for every entry to get objectId. I achive it using find-where operation of backendless
perform a deepsave for every entry passing along the object id if found from before
On first run, every backendless operation takes around 300 to 500 msec
On second run, every backbedless operation takes aprund 80msec
Onbviously for my test, I am using 8 entries only and still wouldnt complete in 5 sec (sometimes it does )
Anything can be optimized here ? Am I using wrong call ?
check if every shelf id exists in data base for every entry to get objectId. I achive it using find-where operation of backendless -check if every product id exists in data base for every entry to get objectId. I achive it using find-where operation of backendless
Make sure you do it with one call
perform a deepsave for every entry passing along the object id if found from before
This probably takes a lot of time. Instead of deep save you can use transaction, it will be faster and will be done with one call. But it will be pretty expensive. Other way is just to call save method in async way. And here you will have to increase your budget because you will have a higher rate limit.
One more option is to have several timers during an hour, and you take less records for one timer execution, in this way you will distribute the operation in time.
When you get the product and shelf data from the external source, do they come with some object identifiers? If so, you could use that data as objectId values when saving products/shelves in Backendless. This will completely remove the step of looking up objectId, and subsequently, if you use the upsert operation, the data will either be created or updated.
Additionally, I recommend implementing a different approach that will let you scale the system regardless of the volume of data returned from the external source and the time it takes to process all the returned entities. The approach evolves around building a queue that contains any unprocessed data entries from the external system.
Your current logic, as it runs once or twice a day, would be in a timer. All it needs to do is fetch the data and save it in Backendless. You could use a simple data table or a Hive List storage. That table (or Hive list) is your queue.
There is a second timer responsible for processing data from the queue. The timer would need to run more frequently (perhaps every minute or two). This timer would need to take one object from the queue and process it according to your insertion (or update) algorithm. You could implement it where it takes more than one object to clear the queue faster.
The benefits of this approach are quite significant. The separation of responsibilities of fetching and persistence frees you up from the necessity to squeeze the entire cycle in 5 seconds and that alone is huge.
Thank you guys @sergey.kuk for suggestion 1, what do you mean by do it in one call? You mean transaction api? Because they are different tables and are different records
@mark-piller this is actually pretty good suggestion around system limitation with queue and definitely sounds like an option to consider to spread the process. However i have one question about your upset suggestion. I know that upsert operation does add/update in one shot. However as far as i know , i cant add 2 records in 2 different tables with one upsert call. Unless I am missing something, I would need to upsert first table, then upsert the second table , then do a relationship set call. Thats 3 calls. Correct ?
I didnt know that I can add different objectId to the table. I thought it has to be autogenerated by the system. Thanks for that suggestion . Will look into that .