Help with external API save to database through timer/business logic

OK, the first step was resolved, it turned out that a new problem arose, because I used a timer block to retrieve data every hour and I used a bulk create block, so every hour the incoming data will be duplicated. what I hope is that only new data will be collected every hour, excluding data that has already been retrieved. What should I do?

Use bulk upsert.

Hereā€™s info about the upsert operation:

IMPORTANT; You will need to rename the ā€œIdā€ property to ā€œobjectIdā€ before objects are upserted into the database

Thanks again Mark, Iā€™ve managed to fetch data from external API from time to time easily.

Hi @Mark_Piller,
Previously, I had successfully implemented a bulk upsert to retrieve data from an external API but it was limited to 100 objects. My question is, is there a way to retrieve more than 100 data from the external API?
thanks for your advice.

Hi, @Iant

You need to have a loop where you can load 100 records with pageOffset until you load all of them. you Can take a look at how itā€™s implemented using Codeless:

Regards,
Marina

Alternatively, you could use a function published to our marketplace that already does it for you:

Hi guys, thanks for your help, anyway Iā€™ve tried both of your suggestions aboveā€¦ As for the installing logic function, Iā€™ve done it but Iā€™m confused about how to apply it because I canā€™t find it in the list of functions. (in the timer logic editor) As for what @Marina suggested I have also tried applying it to my case (upsert data from an external database) but something was missing when connecting pageOffset (data type number could not be connected) and Iā€™m also not sure what i do. Most likely I misunderstood the conceptā€¦
Please helpā€¦
here is what i have done

I thought that the code block that I built earlier would not possibly work. Instead, I tried the following codeā€¦ but it didnā€™t work too

Hello @Iant

What exactly does not work? Some error? I do not know what do you expect.
You should debug your code with a print block.
I recommend do it in UI-Builder. It is convenient for me to do this because there is a developer panel (F12 for chrome) the select the ā€œConsoleā€ tab, this is where your logs from print will appear.
Add component a ā€œbuttonā€ to the page, and add ā€œonClickā€ logic to the ā€œbuttonā€. In ā€œonClickā€ logic run your func.

Debug:

  1. Before the cycle add a print block with ā€œdataBaruā€ (before the green block)
    In the cycle do not use block ā€œin list ā€¦ addā€ with ā€œdataBaruā€. Just save the result of ā€œLoad Table objectsā€ in the new variable ā€œloadedObjectsā€
  2. After the cycle add the ā€œprintā€ block with ā€œloadedObjectsā€
  3. Add the ā€œprintā€ block with the block ā€œIn List (dataBaru) add (loadedObjects)ā€

Also, the ā€œBulk Upsertā€ returns a list of ā€œobjectIdā€

Regards

Hi @Victor thanks for your advice, I did what you recommended but havenā€™t been able to fetch data for more than 100 data objects and store them in a table, but I can already see the data being sent from externalAPI via the payload tab as shown in the following screenshot. What should I do?