Help with external API save to database through timer/business logic

[Iant]

OK, the first step was resolved, it turned out that a new problem arose, because I used a timer block to retrieve data every hour and I used a bulk create block, so every hour the incoming data will be duplicated. what I hope is that only new data will be collected every hour, excluding data that has already been retrieved. What should I do?

[mark-piller]

Use bulk upsert.

Here’s info about the upsert operation:

Object Upsert API | Backendless Database Training Course (pt. 30)

IMPORTANT; You will need to rename the “Id” property to “objectId” before objects are upserted into the database

[Iant]

Thanks again Mark, I’ve managed to fetch data from external API from time to time easily.

[Iant]

Hi @Mark_Piller,
Previously, I had successfully implemented a bulk upsert to retrieve data from an external API but it was limited to 100 objects. My question is, is there a way to retrieve more than 100 data from the external API?
thanks for your advice.

[Marina.Kan]

Hi, @Iant

You need to have a loop where you can load 100 records with pageOffset until you load all of them. you Can take a look at how it’s implemented using Codeless:

How to set pageSIze > 100 in the "Load Table Objects" block

Codeless block “Load Table Objects” doesn’t have the ability to load more the 100 objects. To load data with “page size” > 100 you can create custom “loadTableObject” function. It works like the default block. But keep in mind that for every 100 pages there is a separate request. (There will be 5 requests for pageSize = 430) Notice: On the “Managed” or “Pro” plan you can increase page size limit [Codeless - BL - dsdfsd - Backendless 2021-12-01 12-48-31] [Codeless - BL - dsdfsd - Backendl…

Regards,
Marina

[mark-piller]

Alternatively, you could use a function published to our marketplace that already does it for you:

[Iant]

Hi guys, thanks for your help, anyway I’ve tried both of your suggestions above… As for the installing logic function, I’ve done it but I’m confused about how to apply it because I can’t find it in the list of functions. (in the timer logic editor) As for what @Marina suggested I have also tried applying it to my case (upsert data from an external database) but something was missing when connecting pageOffset (data type number could not be connected) and I’m also not sure what i do. Most likely I misunderstood the concept…
Please help…
here is what i have done

[Iant]

I thought that the code block that I built earlier would not possibly work. Instead, I tried the following code… but it didn’t work too

[viktor.liablin]

Hello @Iant

What exactly does not work? Some error? I do not know what do you expect.
You should debug your code with a print block.
I recommend do it in UI-Builder. It is convenient for me to do this because there is a developer panel (F12 for chrome) the select the “Console” tab, this is where your logs from print will appear.
Add component a “button” to the page, and add “onClick” logic to the “button”. In “onClick” logic run your func.

Debug:

1. Before the cycle add a print block with “dataBaru” (before the green block)
   In the cycle do not use block “in list … add” with “dataBaru”. Just save the result of “Load Table objects” in the new variable “loadedObjects”
2. After the cycle add the “print” block with “loadedObjects”
3. Add the “print” block with the block “In List (dataBaru) add (loadedObjects)”

Also, the “Bulk Upsert” returns a list of “objectId”

Regards

[Iant]

Hi @Victor thanks for your advice, I did what you recommended but haven’t been able to fetch data for more than 100 data objects and store them in a table, but I can already see the data being sent from externalAPI via the payload tab as shown in the following screenshot. What should I do?