Best strategy to process a large list while avoiding execution timeout

Server Code Codeless
1

I have a use case which requires me to process a large list of data on an ad hoc basis.

If I just trigger a method to loop through the list, it is certain to cause an execution timeout even tgough I have purchased multiple execution time extensions. I estimate that the list takes about 10 minutes to process so this would not be advisable.

The simple strategy would be to set a timer to gradually process the list. However, since this use case just requires a 10 minute workload approximately once per week, the timer would be running empty the rest of the time.

The other option I have though of is to bulk create entries in a database table with the unique id’s of the list and then swr an after create event handler that will process each object in the list. My assumption is that each trigger of the event handler would run in parallel and not sequentially thus avoiding the execution timeout.

Is this second option okay to use? Is there another strategy you would advise? Is there a way to activate a timer only as long as a certain list is not empty or something like that?

2

I can advise a third variant – also based on timers.

  • you can create the timer with quite complex schedule using cron syntax;
  • e.g. it will be run 10-15 times but only in one day of week;
  • you divide your data into several parts which can be processed in a 1-2 minutes;
  • yes, a few launches can be idle, but there is nothing to worry about;

It was not clear for me where is the list stored.

3

Thanks, this could work.
The list is initially stored on a Google Sheet. We have logic that pulls from a Google Sheet into a variable.

image
image866×168 24.9 KB

But just checking, the option of generated parallel event handler triggers is not recommended because it creates peak activity? Or is there another reason?

4

Is it possible to activate/stop a timer via API?

5

because it creates peak activity?

Yes it could be so. Also it may slowdown the work with your table.
I thought about your problem as a bulk processing and try to solve it the most efficient way.
It may be convenient to store the data into some temporary table beforehand.

Is it possible to activate/stop a timer via API?

It is a pity, but for now it can be done only through the web-console.
That is why i recommend to use a cron type timers.