Very long response times for easy queries

Data Service
1

Dear Support,

today we are experiencing very long response times for quite easy queries (i.e. no complex selection, no large volumes).

Here you can see that a simple query for 8 objects took 1.1 Minutes of time: https://s3-eu-west-1.amazonaws.com/uploads-eu.hipchat.com/62625/433014/iBIk0dwwM33dqne/upload.png</img>

At other times, the exact same query took only about 1.5 Seconds - we even ran the same query using curl in the command line and these are the results: backendless query times.txt
Columns are curl’s values “time to connect”, “time of transfer”, “time total”, all in seconds.
You can see that the same query runs fine about 10 times, then takes a little over a minute, then runs fine again for some times and then takes forever again. After that it even gets worse. All tests were run around 7 PM CET Today, one at a time, 1 sec delay between two runs. We received notes about slow responses early today and they continued throughout the day, so this is not a temporary situation.

Can you please tell us why this is happening and what we can do about it? Our customers are (of course) very angry about this, as a single long-running query may slow you down for over a minute - this is unacceptable.

Best regards,
Fabian Off

2

Hi Fabian,

Do you experience any slow downs when you browse the data in the console?

Regards,
Mark

3

Dear Mark,

yes, some clicks in the console take as long as in our app.
No fancy stats here, but this is what Charles tells me: https://s3-eu-west-1.amazonaws.com/uploads-eu.hipchat.com/62625/433013/Pu8S05yEK04eqpc/Screen%20Shot%202014-07-22%20at%2019.14.45.png</img>

Same thing for loading pages in the list view.

Best regards,
Fabian

4

We have made a few changes. Could you please check now?

5

Dear Mark,

it seems to work better now, though I still experience some hiccups in the console (~12 secs for a page of 15 elements).
Could you please tell me what caused these slow responses, “a few changes” is a bit vague :wink:

Best regards,
Fabian

6

We have restarted the web tier - the nginx servers. We noticed that occasionally they cause the behavior which you observed. We will be investigating it further as the root cause is still unknown, but we strongly suspect that it is something in nginx.

Do you see high latency in console in a specific area or is it random?

Regards,
Mark

7

Dear Mark,

they seem pretty random, but always data-queries, never static resources.

Best regards,
Fabian