Error messages on LIVE Play servers

General
1

Restarted taskman, play services and consolejs on each server already and still getting a ton of these
Address: 10.0.1.92
Time: 2018-05-22 18:10:24.376 UTC
Message: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Application ID: BC15C3B1-2D4C-C9F2-FFA0-12C7235B7B00

Internal Server Error ID: 5CC610A8-58A0-5D69-FFCB-F9A9B9DDF800

java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at com.backendless.datamodel.DefaultUserDataMapper.getDefaultUser(DefaultUserDataMapper.java:83)
at com.backendless.datamodel.DefaultUserDataMapper.getDefaultUserIdFromDB(DefaultUserDataMapper.java:64)
at com.backendless.datamodel.DefaultUserDataMapper.getDefaultUserId(DefaultUserDataMapper.java:52)
at com.backendless.old.security.Authorization.getUserId(Authorization.java:105)
at com.backendless.PlayContext.getUserId(PlayContext.java:15)
at com.backendless.context.Context.getUserId(Context.java:93)
at com.backendless.servercode.codeModel.ModelCodeExtender.updateRunnerContext(ModelCodeExtender.java:50)
at com.backendless.servercode.codeModel.ModelCodeExtender.getContext(ModelCodeExtender.java:73)
at com.backendless.services.persistence.common.PersistenceCustomLogicDecorator.findById(PersistenceCustomLogicDecorator.java:187)
at com.backendless.services.persistence.common.PersistenceCustomLogicDecorator.findById(PersistenceCustomLogicDecorator.java:56)
at com.backendless.services.persistence.common.PersistenceInterfaceAdapter.findById(PersistenceInterfaceAdapter.java:113)
at controllers.Persistence.findById(Persistence.java:257)
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$423$$anonfun$apply$423.apply(Routes.scala:12340)
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$423$$anonfun$apply$423.apply(Routes.scala:12340)
at play.core.routing.HandlerInvokerFactory$$anon$4.resultCall(HandlerInvoker.scala:157)
at play.core.routing.HandlerInvokerFactory$$anon$4.resultCall(HandlerInvoker.scala:156)
at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$14$$anon$3$$anon$1.invocation(HandlerInvoker.scala:136)
at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:73)
at play.http.HttpRequestHandler$1.call(HttpRequestHandler.java:54)
at com.backendless.old.security.OriginAction.call(OriginAction.java:39)
at com.backendless.old.security.ResumeSessionAction.call(ResumeSessionAction.java:31)
at com.backendless.old.security.ValidateApplication.call(ValidateApplication.java:57)
at com.backendless.old.security.ApplicationExistenceAction.call(ApplicationExistenceAction.java:32)
at com.backendless.old.security.ConsiderApiCallAction.call(ConsiderApiCallAction.java:29)
at com.backendless.TimeOutAction.call(TimeOutAction.java:26)
at com.backendless.SimultaneousApiCallLimitAction.call(SimultaneousApiCallLimitAction.java:22)
at com.backendless.CloseEntityManagerAction.call(CloseEntityManagerAction.java:36)
at play.core.j.JavaAction$$anonfun$7.apply(JavaAction.scala:108)
at play.core.j.JavaAction$$anonfun$7.apply(JavaAction.scala:108)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:70)
at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:48)
at scala.concurrent.impl.Future$.apply(Future.scala:31)
at scala.concurrent.Future$.apply(Future.scala:494)
at play.core.j.JavaAction.apply(JavaAction.scala:108)
at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:111)
at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:110)
at scala.Option.map(Option.scala:146)
at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at com.backendless.datamodel.DefaultUserDataMapper.getDefaultUser(DefaultUserDataMapper.java:79)
… 57 more

2

Were there any changes made in the system such as?:

    backup/restore adding new servers adding new code runners
3

restore of the database was done yesterday. All that was done today was a restart of services and thats when this error started

4

when attempting to open business logic in Console we get the Backendless encountered an issue and internal trouble ticket was created.

5

by looking at the stack trace the error could occur only if the system user (used internally) is deleted from a backendless system table.

Did you restart the system yesterday after the database restore had been performed?
Where does the backup of the database come from?

6

no didnt restart the system yesterday as everything was working up until I restart taskman, consolejs, and play services today.

The database backup came from a RDS mysqldump of the application ID database

7

It looks like it was not the restart that caused the problem, it was the problematic backup which took effect when the system got restarted.

Could you please check if the following query in the “User” table in the application’s database returns any results:
SELECT * FROM .User where default= 1;

8
9

Just to confirm, this is the database you have restored yesterday?

10

Yes

11

This is our LIVE application and all logins are now blocked as no API calls are working. I have invited you to the app.

12

i understand. Out of interest, what was the purpose of performing the database restore?

13

Please run the following and report the result to us (this is a view in the database)

SELECT * FROM DefaultUser;

14

During some automated tests, the automated tests were were accidentally pointed at this app. When the tests finished the data tables in the console appeared to be empty and were showing “0” when viewing through the console Data section.

15

Run against the app database or main_backendless?

16

the app database.

17

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)}
span.s1 {font-variant-ligatures: no-common-ligatures}

mysql> SELECT * FROM DefaultUser;

Empty set (0.01 sec)

18

This is very strange, as the view runs almost identical select query against the “User” table as you executed earlier (you can see the View definition).

Please open the view definition and execute the same query as you will see there directly against the app database.

19

How does one open the view definition from mysql CLI. There is no GUI access to this database instance

20

perhaps this will help?

https://dev.mysql.com/doc/refman/8.0/en/show-create-view.html