Diagnosing very long (20-50 second) network API calls

I make calls to an external API. They fine 95% of the time, and then seemingly randomly, it takes 20-50 seconds to complete. I have simple timer checkpoints around the calls like this -

The thing that is confusing me is the logs on the other side of this call (I can see when the API is called) don’t show the delay. They show the call was made, and the next call in the code happens a second later. Is it possible there is a problem closing the connection or something like that?

I’m looking for suggestions on how to troubleshoot this.

Thanks,
Tim

Hello @Tim_Jones

Could you tell me if it still reproducing for you?
If yes, tell me your app id, service/method name and we will investigate why it happens.

Thanks, @Dima. I am still having trouble.

App Id: EEE25B20-17FA-97DD-FF29-EC45A5072A00

Cloud code function: addMicrostore

Tim

@Tim_Jones can I call this function? And if yes, is there exists some additional instruction that I must follow?

That function will add a new microstore every time you call it, which is not the best.

There is an update function that calls the same API and also has similar issues. I created a quick test invocation for you. The test service has a /testmicrostorefunctions method you can call.

If you’d like to test the addMicrostore function, I can set up a test service and delete the created microstores later.

Thanks,
Tim

Unfortunately, I can’t give a direct answer it's because of X. As your function is complicated there could be various roots of issues on different levels, from your code to DB operation.

I could advise you to create a logger functionality that will help you to get statistics on what part of the code takes the biggest part of the time.

You can create an object that contains an operation name(for example - looking for records in DB) and operation start/end time. That object should be written somewhere.

When we get a couple of records with statistics (including long responses that you randomly have), we can analyze them and get the answer why it happened.

I have that for the addMicrostore Function but not the updateMicrostore Function. The logs looks like this -

Time check details:
{“checkPoint”:1,“elapsedTime”:0}
{“checkPoint”:2,“elapsedTime”:78}
{“checkPoint”:3,“elapsedTime”:27493}
{“checkPoint”:4,“elapsedTime”:27493}
{“checkPoint”:5,“elapsedTime”:27611}

The code between checkpoints 2 and 3 is what I posted above. That is always where the delay is from and where I’m stuck.

Tim

Request by path /api/v1/mictostores it is Backendless service?

No. It is an external API.

Hello @Tim_Jones

Perhaps the request to the /api/v1/mictostores takes a long time
Backendless API Methods run in the cloud. We see requests to the cloud, but we don’t see requests to the /api/v1/mictostores.
Can you create a simple example of this requst:


in the ui-builder?
I’m having a hard time doing this myself as there are a lot of variables used for this request.
In the ui-builder, it will be easier for me to debug because the code is run locally.

Regards

It could be the API is slow, but as I mentioned, the logs on the other side of the call don’t show it being slow. They show the subsequent calls being one second apart but my logging shows 20-50 seconds. So that is why I’m wondering if there is an issue with backendless closing the call or something.

The behavior is random, unfortunately. Most calls are fast, but I get a 20-50 second call time every once in a while.

I built a test page for you. It is in the default container. The page name is testMicrostore.

Thank you!!

Hello @Tim_Jones!

Could you please tell me if this issue is still reproducing for you?
I tried reproducing it on the testMicrostore page, but I did not have any requests for this duration.

Regards,
Alexander

Hi @Alexander_Pavelko,

I am still seeing slow calls in the logs. It is random, as far as I can tell.

Tim

Hi Tim,

I went through this thread trying to understand what’s going on. Is the issue that there is a slow call to a Backendless API or an external service? If the former, could you please point out which API it is?

Regards,
Mark

Hi @mark-piller,

It’s an external API. What I’m asking for help with is making sure there isn’t anything on the backendless side. The external API logs don’t show the slow calls, but my time logging in Backendless does so I’m confused.

Thanks,
Tim

Try making that external API call in Postman or “Advanced REST Client”. These tools should show time spent on establishing a connection, transport time, etc. The logs from the external service may show just the invocation time but may not account for socket allocation, SSL handshake, response serialization, etc. All of these contribute to the overall round-trip time.

Hope this helps.

Regards,
Mark

I’ve made test calls in Postman to figure out the issue. As it’s intermittent, it’s hard to have it happen, let alone figure it out.

I was you (the support team) might guess what might be causing the issue.

As I said, I’m grasping at straws here. The slow calls are causing problems and look bad to the customer, but I have no idea why it’s happening or how to fix it.

Tim

What we see is that you make external call and it takes time. We dispatch the call the instant the logic says to do it. The problem to me appears to be entirely on the side of the service itself. Since the problem is random, try creating a postman script to do the calls in a loop, perhaps you will be able to catch it.

That is a good idea. Thanks, @mark-piller!