[URGENT] Very slow response time today from the app, web, and console + Bad request errors

hello team,
I am experiencing very slow response times today on backendless, making the platform almost unusable both from the client and on the online console (we are getting emails now that users are even unable to register an account, probably due to the very slow response)

it can take a few minutes to do operations that normally I would do in seconds, can anyone please check what is going on?

app ID: 9088AF67-1648-1726-FF09-A51862BF1A00 (backendless 3)

thank you for your help

EDIT: please check this ASAP as users are buying the app and are unable to register, you can imagine how that is going…

EDIT2: all cloud functions are also not working, meaning we have also all the issues reported here with bad requests: Urgent: bad request (in production, prevents users from purchasing our product)

please somebody reply

1 Like

Hi @Alessandra_Saviotti

We’ll check it out
The answer will be as soon as possible.


1 Like

any updates? this is becoming a big problem

we have been down for 5 hours already

this is a production system and is causing a lot of trouble to paying customers, it needs to be resolved ASAP

Hello @Alessandra_Saviotti

we are still working on the issue. It is our priority task. We will notify you as soon as we will have any solution

Hello, @Alessandra_Saviotti

I apologize for the delay, but I had to collect the necessary data and check everything in detail.

We added resources twice for db servers as there should be. But this will not solve the problem. Now there are many slow queries against a large table in you db. These requests greatly slow down the overall operation.
The main problem is that one of the tables in your database is very large and it is not optimized; it may contain a lot of deleted and fragmented data.

We can run the table optimization but it may block access to it for some time, given the size of the table it may take 3-5 hours.

We need your consent for these actions.

thank you for the update

would this block everything or only this table?

either way, right now nothing is usable, so we will need to go on with this optimization operation

please proceed and keep me posted, thank you


This operation will lock only the DailySummaryNoCoreData table.
But all requests in which this table appears will also be blocked and based on the fact that this table is architecturally important for your software product, I predict that the service will be completely blocked or will respond slowly.
The optimization operation as a whole will be difficult because a large volume of data will be manipulated. There is also no guarantee that the optimization will be successful and there will be enough memory.

I will start the optimization process and let you know how the process is progressing.

I understand, but what choice do we have?

as an alternative, would it be possible for you to delete entries collected prior to 2019 (so we get the table smaller). We do not want to lose anything collected in the past 3 years, but we could drop what was collected earlier if you had a way to do that (using the “created” field in that table)

@Alessandra_Saviotti You should have cleared that table if you didn’t need that data.
Now it will be more difficult to do.
I will estimate how much data will need to be deleted and let you know.

the table is needed, but I am thinking about ways to solve the problem here, since right now you are telling me that we are screwed and nothing will work

older data could therefore be deleted, if that’s something you can do more easily than the “optimization”

can you do that? it would be fine if you could delete data collected prior to 2020. I cannot do that myself as the operations are really slow or not working (and I could only delete very few entries at the time)

also: how is it even possible that yesterday everything was perfect and now nothing works? very little has changed in these 24 hours, this makes no sense

cc @mark-piller

So, at this point, I understand that we can delete data older than 2019 and after that try to optimize the table? I think such actions should lead us to success.
Started making changes in table.

1 Like

thank you @Marian_Koreniuk, let’s go with that, delete everything before 2019, and if it is not enough we will think about it, then also the optimizations

keep us posted on progress please


I made some optimization of the table and I see that the work has improved somewhat, and backups of the data that will be deleted are being made. 5.2 million records will be deleted. I will inform you further about the removal. At the moment, is there any improvement in your requests?

1 Like

great thank you, it seems to be working now

just a quick follow up, if we delete also all 2019, do we gain a lot more?

I think so, it makes sense, for 2019 there are approximately 3 million records. This way we can delete more than 7 million records.

So it will speed it up, but I would recommend to look at the architecture of your software product and to separate the generation into a separate table, it would give better results.

thank you Marian, we have been thinking about this and do plan to make changes in the future

let’s remove also 2019 then so we should be good for a while

@Marian_Koreniuk can you double-check this as we seem to be experiencing some delays again today, while yesterday after resolving the issues it seemed all good. Thank you

edit: we are down now again, please help

edit2: now it is working again, please if you have time check that there isn’t anything off as we cannot have these issues on a daily basis, thank you

Hello, @Alessandra_Saviotti.
We solved only part of the problem yesterday. Data is deleted in portions, we cannot delete a large volume at once, this greatly blocks the work of all databases in general. Removal will continue today. In a few hours, will give an estimate of how long it will take for complete removal. We see that there are delays in the operation of the service and are doing everything possible.

1 Like

thank you for the explanation, the Users table seems unresponsive now, just flagging it in case something happened, as this prevents people from registering accounts after downloading the (paid) app.