Inconsistent connection to Backendless servers?

milen-marinov · April 18, 2022, 10:56am

In my app I started logging all handleFault events from Backendless SDK and noticed an unexpectedly high (12K events in a week for an app with ~4K monthly users) number of connection problems to Backendless. (The app checks for internet connectivity before attempting network requests so the problem is not lack of internet altogether)
There are several types of errors logged:
1.

BackendlessFault{ code: ‘Internal client exception’, message: ‘Unable to resolve host “api.backendless.com”: No address associated with hostname’, detail: 'java.net.UnknownHostException: Unable to resolve host “api.backendless.com”: No address associated with hostname at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:156) at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103) at java.net.InetAddress.getAllByName(InetAddress.java:1152) at com.android.okhttp.Dns$1.lookup(Dns.java:41) at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178) at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144) at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86) at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:176) at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:128) at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:97) at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:289) at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:232) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262) at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219) at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30) at weborb.client.ioEngine.HttpIOEngine.send(HttpIOEngine.java:195) at weborb.client.ioEngine.HttpIOEngine.invoke(HttpIOEngine.java:140) at weborb.client.WeborbClient.invoke(WeborbClient.java:138) at com.backendless.Invoker.invokeSync(Invoker.java:95) at com.backendless.Inv<truncated: 732 chars>

BackendlessFault{ code: ‘Internal client exception’, message: ‘Read error: ssl=0x73359f1900: I/O error during system call, Software caused connection abort’, detail: 'javax.net.ssl.SSLException: Read error: ssl=0x73359f1900: I/O error during system call, Software caused connection abort at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method) at com.android.org.conscrypt.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:741) at com.android.okhttp.okio.Okio$2.read(Okio.java:136) at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:211) at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at com.android.okhttp.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.android.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.android.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:471) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:407) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaders(HttpURLConnectionImpl.java:163) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaderFields(HttpURLConnectionImpl.java:223) at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getHeaderFields(DelegatingHttpsURLConnection.java:178) at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getHeaderFields(Unknown Source:0) at weborb.client.ioEngine.HttpIOEngine.storeCookies(HttpIOEngine.java:93) at weborb.client.ioEngine.HttpIOEngine.send(HttpIOEngine.java:199) at weborb.client.ioEngine.HttpIOEngine.invoke(HttpIOEngine.java:140) at weborb.client.WeborbC<truncated: 376 chars>

BackendlessFault{ code: ‘Internal client exception’, message: ‘timeout’, detail: 'java.net.SocketTimeoutException: timeout at com.android.okhttp.okio.Okio$3.newTimeoutException(Okio.java:225) at com.android.okhttp.okio.AsyncTimeout.exit(AsyncTimeout.java:263) at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:217) at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:317) at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:311) at com.android.okhttp.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:207) at com.android.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:395) at com.android.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:146) at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:900) at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:772) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:493) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:429) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaders(HttpURLConnectionImpl.java:171) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaderFields(HttpURLConnectionImpl.java:231) at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getHeaderFields(DelegatingHttpsURLConnection.java:179) at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getHeaderFields(HttpsURLConnectionImpl.java:30) at weborb.client.ioEngine.HttpIOEngine.storeCookies(HttpIOEngine.java:93) at weborb.client.ioEngine.HttpIOEngine.send(HttpIOEngine.java:199) at weborb.client.ioEngine.HttpIOEngine.invoke(HttpIOEngine.java:140) at weborb.client.WeborbClient.invoke(WeborbClient.java:138) at com.backendless.Invoker.invokeSync(Invoker.java:95) at com.backendless.Invoker$1.run(Invoker.java:71) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at<truncated: 459 chars>

BackendlessFault{ code: ‘Internal client exception’, message: ‘SSL handshake timed out’, detail: 'java.net.SocketTimeoutException: SSL handshake timed out at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method) at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:387) at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:226) at com.android.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:196) at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:153) at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:116) at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:186) at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:128) at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:97) at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:289) at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:232) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262) at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219) at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30) at weborb.client.ioEngine.HttpIOEngine.send(HttpIOEngine.java:195) at weborb.client.ioEngine.HttpIOEngine.invoke(HttpIOEngine.java:140) at weborb.client.WeborbClient.invoke(WeborbClient.java:138) at com.backendless.Invoker.invokeSync(Invoker.java:95) at com.backendless.Invoker$1.run(Invoker.java:71) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at jav<truncated: 142 chars>

BackendlessFault{ code: ‘Internal client exception’, message: ‘Failed to connect to api.backendless.com/216.87.89.89:443’, detail: 'java.net.ConnectException: Failed to connect to api.backendless.com/216.87.89.89:443 at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:143) at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:112) at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184) at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126) at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95) at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281) at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:461) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127) at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:258) at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:218) at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(Unknown Source:0) at weborb.client.ioEngine.HttpIOEngine.send(HttpIOEngine.java:195) at weborb.client.ioEngine.HttpIOEngine.invoke(HttpIOEngine.java:140) at weborb.client.WeborbClient.invoke(WeborbClient.java:138) at com.backendless.Invoker.invokeSync(Invoker.java:95) at com.backendless.Invoker$1.run(Invoker.java:71) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636) at java.lang.Thread.run(Thread.java:784) ', extendedData: ‘{}’ }

I guess it’s normal to have a small number of such errors but as I said it looks like all users are experiencing intermittent connection to Backendless. Are you aware of such problems? Are they temporary or consistent?

Marina.Kan · April 18, 2022, 11:49am

Hi, @milen-marinov

I have informed our engineers of your notification. Thanks for contacting us. Reliable and fast operation of Backendless has the highest priority for us. We are looking into the matter.

Regards, Marina

milen-marinov · July 6, 2022, 9:01pm

Is anyone actually going to do anything about this issue? Almost 3 months later nothing has happened and I continue receiving more than 1K failed connection reports DAILY for an app with less thank 4K monthly active users! You are practically forcing us to migrate to Firebase…

mark-piller · July 6, 2022, 9:17pm

Thanks for bumping up this topic. I just reviewed the stack traces and I see that every single issue is related to the internet connection on the client side. I am curious what you think we can actually do here?

If the client-side is having a problem resolving a hostname (issue 1), is there anything we can actually do on the server-side?

If the client-side aborted SSL handshake (issues 2 and 4) for some unknown reason (it could be as silly as the user decided to shut down the app), what can we do about it?

If the client-side is failing to connect (issue 5), this would be a big issue if we heard this from multiple users, but no, we see constant traffic and no one else reports this issue. My assumption is it is something that is specific to the device/network/vpn/additional software running on the device. Without any additional information, we would not be able to do anything.

This leaves issue 3, which again is too vague to be able to do anything. By reading the stack trace, it appears the client timed out while reading the response’s headers. Why it happened, it would be impossible to say definitively without a way to reproduce the problem.

We care about problems and there is a long history of us consistently fixing bugs and improving the service. However, we’re not magicians that can snap fingers and your stack traces go away. We ask to provide information so we can identify an issue and reproduce it. In some cases a stack trace is sufficient. In this particular example, unfortunately, they do not provide any helpful details.

Mark

milen-marinov · July 6, 2022, 9:32pm

As you can see from the above screenshot affected users are 870 for the past month. It’s unlikely that 20% of MAU all have various connection problems. It looks like connection to your servers is unreliable and dropping at different stages. Not sure where your servers are located and whether this is connected with the war in Ukraine as the timing coincides.
As for the lack of other Backendless users complaining - I recently started logging ALL faults from the app, not just crashes, which is not common practice. So there may just be lack of awareness of the issue.
If you tell me what information you need - I’ll gladly provide it!

mark-piller · July 6, 2022, 9:43pm

If that were the case, and with thousands of apps using Backendless on the 24x7 basis, do you think there would be more people bringing up this issue?

You can easily check by going to https://iplocation.io/ and entering api.backendless.com.

Here’s more on the US data center where Backendless runs:

I would disagree with you, everyone who has an app in production does the same.

Have you actually heard from the users complaining?
If so, have you checked what networks they are on?
What geography they are from?
What providers do they use?
Could it be they are coming from some underdeveloped area with a bad Internet connection?