Support Topics Documentation Slack YouTube Blog

Queue 1000 objects into the DB daily

My use case requires me to update 1000 objects from an external source into one of my tables daily.

I am struggling to find an elegant way to do this.

Current logic:

  1. Set a timer to repeat every day, set a counter to 1000.
  2. Set a second timer to repeat every minute. In the second timer, logic sets the counter to decrement from 1000 by 1 and triggers an API method that does the updating
  3. Set a before event handler that blocks the api method when the counter reaches zero

This is not a very elegant method. Is there a better way to update 1000 objects daily, without allowing execution timeouts but remembering where I left off and stopping further code running until the next day?

From my perspective it is better to use bulk update.
So using whereClase you can create the condition which describes what objects to update. E.g. you can use “created” and “updated” fields in order to filter current day objects.

Of course if you need objects to be updated periodically, the timers is a right way to go.

I need the objects to be updated on a daily basis. The main issue I have is that there are 1000 objects and I get an execution timeout.

I understand that with bulk update, I cannot have different values per object property (eg. object 1, property phone: 123 while object 1, property phone: 456). So I cannot use Bulk Update.

Looking through the documentation, I believe I can use a Database Event Handler to simulate a queue with shift and pop.

  1. I have a daily timer that loads an external array into a DB table, then updates the first object in the table, then deletes it.
    eg. Create [1, 2, 3], Update 1, Delete, 1

  2. Add an event handler with Model: default, Category: Database API, Event: remove, Timing: after. The logic will update the next object in the queue and delete it.

This will continue until all objects in the queue are updated/deleted.

However, I am unable to trigger this event when I delete an object from the table.

I have created a simple logic to test it but nothing is printing.


Am I doing something wrong?

If you schedule a timer to run every 10 minutes throughout the day and handle 4 objects at a time, you will be able to process 1000 objects in 24 hours. I believe this is strictly a problem of queue management.

1 Like