Error in business logic after outage

Thanks Oleg. Any advice for the logging that stopped working?

Iā€™ve tried and received the error when you code is invoked in cloud mode.
But for the debug mode it works (try to use latest coderunner with java-11).
For now iā€™m working on the error from cloud.

p.s. I think the loggerā€™s work doesnā€™t related to this problem. It should be something different.

So for me in debug mode, it does not throw an error. However it also does not print anything to console like it should when timer is run, and it also does not write the data to backendless tableā€¦

EDIT:

I am now receiving an error when running the timers in debug mode:

java.net.SocketException: Unexpected end of file from server
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:862)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:859)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1507)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1505)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1504)
	at java.base/java.net.URLConnection.getContent(URLConnection.java:749)
	at com.btcmarketsticker.timers.JSONWorker.getJSONFromURL(JSONWorker.java:45)
	at com.btcmarketsticker.timers.PriceGetterTimer.execute(PriceGetterTimer.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.backendless.coderunner.runtime.task.TimerInvocationTask.runImpl(TimerInvocationTask.java:52)
	at com.backendless.coderunner.runtime.executor.ExtendedRunnable.run(ExtendedRunnable.java:38)
	at java.base/java.lang.Thread.run(Thread.java:834)

EDIT2: sometimes I will also receive this error in debug mode:

javax.net.ssl.SSLException: Cannot allocate new thread. You can remove this limitation by purchasing a function pack in Backendless Marketplace.

at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)

at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1313)

at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:408)

at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)

at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168)

at com.btcmarketsticker.timers.JSONWorker.getJSONFromURL(JSONWorker.java:40)

at com.btcmarketsticker.timers.PriceGetterTimer.execute(PriceGetterTimer.java:61)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at com.backendless.coderunner.runtime.task.TimerInvocationTask.runImpl(TimerInvocationTask.java:52)

at com.backendless.coderunner.runtime.executor.ExtendedRunnable.run(ExtendedRunnable.java:38)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.security.AccessControlException: Cannot allocate new thread. You can remove this limitation by purchasing a function pack in Backendless Marketplace.

at com.backendless.coderunner.runtime.security.CodeRunnerSecurityManager.checkAccess(CodeRunnerSecurityManager.java:67)

at java.base/java.lang.ThreadGroup.checkAccess(ThreadGroup.java:313)

at java.base/java.lang.Thread.<init>(Thread.java:423)

at java.base/java.lang.Thread.<init>(Thread.java:765)

at java.base/sun.security.ssl.TransportContext.finishHandshake(TransportContext.java:610)

at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:758)

at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:659)

at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)

at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Finished.java:1025)

at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Finished.java:885)

at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)

at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:450)

at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:427)

at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)

at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)

at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)

at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)

at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)

... 12 more

javax.net.ssl.SSLException: Cannot allocate new thread. You can remove this limitation by purchasing a function pack in Backendless Marketplace.

at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)

at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)

at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1313)

at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:408)

at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)

at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:100)

at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:80)

at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:724)

at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1608)

at java.base/sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1507)

at java.base/sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1505)

at java.base/java.security.AccessController.doPrivileged(Native Method)

at java.base/java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:795)

at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1504)

at java.base/java.net.URLConnection.getContent(URLConnection.java:749)

at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContent(HttpsURLConnectionImpl.java:430)

at com.btcmarketsticker.timers.JSONWorker.getJSONFromURL(JSONWorker.java:45)

at com.btcmarketsticker.timers.PriceGetterTimer.execute(PriceGetterTimer.java:61)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at com.backendless.coderunner.runtime.task.TimerInvocationTask.runImpl(TimerInvocationTask.java:52)

at com.backendless.coderunner.runtime.executor.ExtendedRunnable.run(ExtendedRunnable.java:38)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.security.AccessControlException: Cannot allocate new thread. You can remove this limitation by purchasing a function pack in Backendless Marketplace.

at com.backendless.coderunner.runtime.security.CodeRunnerSecurityManager.checkAccess(CodeRunnerSecurityManager.java:67)

at java.base/java.lang.ThreadGroup.checkAccess(ThreadGroup.java:313)

at java.base/java.lang.Thread.<init>(Thread.java:423)

at java.base/java.lang.Thread.<init>(Thread.java:765)

at java.base/sun.security.ssl.TransportContext.finishHandshake(TransportContext.java:610)

at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:758)

at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:659)

at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)

at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Finished.java:1025)

at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Finished.java:885)

at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)

at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:450)

at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:427)

at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)

at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)

at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)

at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)

at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)

... 21 more

I have encountered this error before (something like 2 years ago) and got around it by specifically adding the line request.addRequestProperty("Connection","close"); which you can see in the above code sample.

You may try new api for http/s requests. It is more convenient.

      HttpClient httpClient = HttpClient.newHttpClient();
      HttpRequest request = HttpRequest.newBuilder( URI.create(sURL) ).GET().build();
      HttpResponse<String> response = httpClient.send( request, HttpResponse.BodyHandlers.ofString() );

The fix for the old code is almost done. Iā€™m testing it now.

Could you check your code now. Weā€™ve made the update.

Hi Oleg,

That appears to have fixed the issue. However now I receive error that timer execution is terminated due to time taking too long (> 20 seconds with function pack). Custom business logic execution has been terminated because it did not complete in permitted time - 20 seconds

This was never an issue before?

Could it be related to the outer server you make requests to ?

I donā€™t think so, I can make connection to same URLs almost immediately in browser

There was the problem with configuration.
Please, find in the directory with CodeRunner file ā€¦/bin/runner.properties and replace the parameter

system.thread.manipulation.exclude.classes = weborb.cloud.AmazonBillingClient,sun.awt.AppContext$2,java.awt.image.ColorModel$1,com.sun.imageio.stream.StreamCloser$2,sun.java2d.Disposer,sun.net.www.http.KeepAliveCache$1,sun.net.www.protocol.http.HttpURLConnection,com.sun.net.ssl.HttpsURLConnection,sun.net.www.protocol.https.HttpsClient,sun.net.www.http.HttpClient,sun.net.www.http.KeepAliveCache,sun.security.ssl.SupportedGroupsExtension,sun.security.ssl.SupportedGroupsExtension$SupportedGroups,sun.security.ssl.SSLContextImpl,sun.security.ssl.SSLContextImpl$DefaultSSLContext,sun.security.ssl.SSLSocketImpl,jdk.internal.net.http.HttpClientImpl,jdk.internal.net.http.HttpClientImpl$DelegatingExecutor,jdk.internal.net.http.HttpClientImpl$DefaultThreadFactory,jdk.internal.net.http.HttpClientImpl$SelectorManager,jdk.internal.net.http.common.MinimalFuture,jdk.internal.net.http.PlainHttpConnection,jdk.internal.net.http.MultiExchange,jdk.internal.net.http.SocketTube,jdk.internal.net.http.common.SSLTube,jdk.internal.net.http.common.SSLTube$DelegateWrapper,jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper,jdk.internal.net.http.common.SubscriberWrapper,jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher,jdk.internal.net.http.common.SequentialScheduler,jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask,jdk.internal.net.http.common.SequentialScheduler$SchedulableTask,jdk.internal.net.http.common.SSLFlowDelegate,jdk.internal.net.http.common.SSLFlowDelegate$Reader,jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher,sun.net.www.protocol.https.DelegateHttpsURLConnection,sun.nio.ch.EPoll,sun.nio.ch.SelectorImpl

It should help.

In next release these properties will be added automatically.

Hi Oleg,

That is now working in debug and production. Everything is back to normal. Thank you very much.

Is there anything I can do differently in future to avoid outages/issues like this?

Glad to hear that everything is ok.
Also, Mitchell, thank you for the help.
The problem was related to the changes between java-8 and java-11.
It wasnā€™t your fault, so you really couldnā€™t do anything in such a situation.
Once again, thanks for your patience.

p.s. i would recommend to use new java-11 http/s api, itā€™s really convenient.

My pleasure Oleg. I will be sure to try the new API, your example above looks much easier than my current implementation.

Thanks again!