Help with external API save to database through timer/business logic

[Iant]

Thanks for your fast response, actually i also try to extract the list of data from this api https://sheetdb.io/api/v1/vzbboolwx9aot and below is log

[Iant]

i’m sorry i don’t know about print block

[mark-piller]

This is the print block. It logs a message in the log file:

[Iant]

Hi Mark,
Thank you for showing it but I still don’t understand how to use it, I tried like the following image but I’m not sure if i’m doing something right because I don’t know what value to input.

[mark-piller]

You can print anything that will facilitate solving the problem. I don’t quite understand what the problem is, but with the print block you can have a way for the logic you create to report it’s state (variables, service responses, etc) back to you via logging.

[Iant]

Okay mark, I’ll try to explain again the problem I’m having, so I’m trying to periodically fetch data from the external api and store it in a table.
I’ve tried to make codeless logic like above but no data goes to the table except the default backendless column. The External API is https://sheetdb.io/api/v1/vzbb00lwx9aot
and logs looks like this

I hope it helps to find a solution, thank you.

[mark-piller]

The link for the API returns a JSON array. How should the data in that array be mapped to a table in database?

[Iant]

it’s true that the data is in an array, that’s the problem, I’m confused about how to map the data to a table. I need help and advice because I really don’t know about this yet.

[mark-piller]

Well, one possibility is to store each object in the array as a separate object. In that case, the Bulk Create block would do the job:

Bulk Create API | Backendless Database Training Course (pt. 27)

[Iant]

Great, it worked successfully!
Thank again Mark…

[Iant]

OK, the first step was resolved, it turned out that a new problem arose, because I used a timer block to retrieve data every hour and I used a bulk create block, so every hour the incoming data will be duplicated. what I hope is that only new data will be collected every hour, excluding data that has already been retrieved. What should I do?

[mark-piller]

Use bulk upsert.

Here’s info about the upsert operation:

Object Upsert API | Backendless Database Training Course (pt. 30)

IMPORTANT; You will need to rename the “Id” property to “objectId” before objects are upserted into the database

[Iant]

Thanks again Mark, I’ve managed to fetch data from external API from time to time easily.

[Iant]

Hi @Mark_Piller,
Previously, I had successfully implemented a bulk upsert to retrieve data from an external API but it was limited to 100 objects. My question is, is there a way to retrieve more than 100 data from the external API?
thanks for your advice.

[Marina.Kan]

Hi, @Iant

You need to have a loop where you can load 100 records with pageOffset until you load all of them. you Can take a look at how it’s implemented using Codeless:

How to set pageSIze > 100 in the "Load Table Objects" block

Codeless block “Load Table Objects” doesn’t have the ability to load more the 100 objects. To load data with “page size” > 100 you can create custom “loadTableObject” function. It works like the default block. But keep in mind that for every 100 pages there is a separate request. (There will be 5 requests for pageSize = 430) Notice: On the “Managed” or “Pro” plan you can increase page size limit [Codeless - BL - dsdfsd - Backendless 2021-12-01 12-48-31] [Codeless - BL - dsdfsd - Backendl…

Regards,
Marina

[mark-piller]

Alternatively, you could use a function published to our marketplace that already does it for you:

[Iant]

Hi guys, thanks for your help, anyway I’ve tried both of your suggestions above… As for the installing logic function, I’ve done it but I’m confused about how to apply it because I can’t find it in the list of functions. (in the timer logic editor) As for what @Marina suggested I have also tried applying it to my case (upsert data from an external database) but something was missing when connecting pageOffset (data type number could not be connected) and I’m also not sure what i do. Most likely I misunderstood the concept…
Please help…
here is what i have done

[Iant]

I thought that the code block that I built earlier would not possibly work. Instead, I tried the following code… but it didn’t work too

[viktor.liablin]

Hello @Iant

What exactly does not work? Some error? I do not know what do you expect.
You should debug your code with a print block.
I recommend do it in UI-Builder. It is convenient for me to do this because there is a developer panel (F12 for chrome) the select the “Console” tab, this is where your logs from print will appear.
Add component a “button” to the page, and add “onClick” logic to the “button”. In “onClick” logic run your func.

Debug:

1. Before the cycle add a print block with “dataBaru” (before the green block)
   In the cycle do not use block “in list … add” with “dataBaru”. Just save the result of “Load Table objects” in the new variable “loadedObjects”
2. After the cycle add the “print” block with “loadedObjects”
3. Add the “print” block with the block “In List (dataBaru) add (loadedObjects)”

Also, the “Bulk Upsert” returns a list of “objectId”

Regards

[Iant]

Hi @Victor thanks for your advice, I did what you recommended but haven’t been able to fetch data for more than 100 data objects and store them in a table, but I can already see the data being sent from externalAPI via the payload tab as shown in the following screenshot. What should I do?