Backendless down once again for 5 hours on february 16th

hi,
I have received news that once again the service will be down for a very long time period (5 hours) due to your failure to pull this off the last time you tried (which also casued huge damage to our business, and I won’t start listing the way to high number of times these issues happened in the past 2 months alone).

Anyhow, since this time at least you had the decency to provide this information a bit in advance, I am asking for the following (not that I expect it to be of any use, considering that most of the times your service does not work the way you say it does, like when you blocked us completely on January 1st because of a failed payment, despite the fact that you keep denying it and saying that you do not do so for up to 5 days):

  • how do you recommend that in the apps we handle the fact that you are offline? Obviously people will want to use the service, and even worse, people will buy the app and will be stuck there for hours. Do you return a specific error that can be handled so that we can provide useful feedback to the user? Please note that the last times this happened we got all sort of different errors, hence it does not seem that you do this properly and consistently.
  • please provide code in Objective C and Java showing a simple attempt to login and handling the fact that the server is down, plus a way for us to test this before the 16th, so that we can at least try to make changes in the app to inform the user and avoid another total fuckup, with negative reviews and hundreds of emails from angry customers

Considering that we have been paying for this service for years, and that the damage recently has been difficult to contain due to how unreliable it became, please try at least this time to make sure we can be prepared (and that we can test it). Provide code and instructions and a way to test the whole thing ASAP

This is of course for backendless 3.

We’re preparing this information for you.

Regards,
Mark

Here’s the process which can be implemented to check if the migration is in progress: There are two endpoints which can be checked: (1) a live route indicating whether V3 migration is in progress and (2) a test route which is a simulation for testing purposes:

Live route:

GET https://api.backendless.com/v3migrationstatus  

While the 3.x migration is in progress, the route will return HTTP code 503 with the following in the response payload:

{  
  "message" : "The 3.x cluster is being migrated",  
  "code": 1999  
}

When migration is not in progress, the route returns HTTP code 200 without any payload.

Test route:

GET https://api.backendless.com/v3migrationstatus-test  

the route always returns HTTP 503 with the following in the response payload:

{
“message” : “The 3.x cluster is being migrated”,
“code”: 1999
}
These routes are being put in place today. We will notify you when they are ready.

Regards,
Mark

thank you, we will be trying this and let you know in case of issues.

Both routes are now in place and can be tested from your app.

Regards,
Mark

alright, we have integrated this hoping it all works out on the 16th, our service does not require to be online (at least the apps), so the idea is to check the code, and if it is 1999, we pretend to be offline and the user can go ahead and take the measurement. If there is no code or another code, then it means there is no migration going on, and we go through the usual automatic login. This should avoid the login screen to be displayed because backendless is offline, which is normally what causes issues on our end (as the users are stuck outside the app). Makes sense?

I have a couple of follow up points / questions:

  • we will enable this only on the 16th, as otherwise there is an extra call that delays regular execution, please if anything changes in your schedule let me know asap as this is hardcoded in the app update that we will send out later this week
  • is there any chance that the migration on your end lasts less than the scheduled 5 hours?
  • can you confirm that these routes will be available before, during and after the migration? (like several days, as they are now part of our login logic)
  • please make sure that there is no gap in the data, last time that this was tried at a certain point we had lost some months of data, then it was recovered, but I am just mentioning it as this is really important

Finally on an unrelated note, I have been unable to export ‘all my tables’ in the past 2-3 months, I have mentioned it several times but have not gotten any working solution, it would be great if you could look into this. The export email says there was an error but does not provide additional details, maybe you have better log files.

Thank you

I assume you check for HTTP 200/503 first.

  • is there any chance that the migration on your end lasts less than the scheduled 5 hours?

Yes, it is possible the migration will be done sooner.

  • can you confirm that these routes will be available before, during and after the migration? (like several days, as they are now part of our login logic)

Confirmed.

  • please make sure that there is no gap in the data, last time that this was tried at a certain point we had lost some months of data, then it was recovered, but I am just mentioning it as this is really important

Confirmed.

Finally on an unrelated note, I have been unable to export ‘all my tables’ in the past 2-3 months, I have mentioned it several times but have not gotten any working solution, it would be great if you could look into this. The export email says there was an error but does not provide additional details, maybe you have better log files.

I do not remember seeing a support topic for it. Please let me know the link if you have it. Have you checked the “export” directory in the file storage of Backendless? There should be a log file in there with additional details.

Regards,
Mark

here is the message I get (I’ve only removed the application name, the rest is what I receive:

" Backendless finished export your data unsuccessfully for application: ** application name ** .
The Export log located in File Service in export directory.
You can download exported zip file from File-Service.
Download link: null

See export log bellow:

Sorry, we have unknown problem when attaching log file to the email"

The log file in console, under export, is however not updated (last edited in January, when I was able to export a single table), hence I have no other information on my side, not sure if you can see anything extra

hi Mark, it seems that both methods are equivalent (either checking that it returns 200 or 503 or checking that it returns 1999 or it doesn’t), meaning that there is no case in which the two would not match. Or am I missing somethings?

Thank you for your help

Hi Mark,

I understand this topic goes for V3 only. That said, I remember last time there was a big update for V3, there was an update for V5 as well - and my colleague reported being unable to login for a couple of hours through V5.

While we didn’t get any notification for Backendless V5 being down on Feb 16 (meaning there might not be an update for us at all), I wonder to avoid any unhandled scenarios, should such endpoint exist and be tracked for us using V5 as well?

Thanks,
Justinas

Hi @Justinas_Grazulis,

Backendless V5 has already migrated. So there is no route for V5 and you should not do anything special for this day.

Hi Sergey,

That is good news, thanks for the update!

please let me know when you ready this, thank you @mark-piller

When you say “both methods”, I do not quite understand what you refer to. There is only one route to check the status, the one shown below:

GET https://api.backendless.com/v3migrationstatus

The other route is only for testing purposes, so you can test your app with. The route above can return either HTTP 200 or HTTP 503, just like it is described here.

The

The log file should not be updated. Backendless creates a new log file for every export. I do not have access to your app, so I cannot check myself. Do you possibly have more log files that fit on a single page of the File Browser in Backendless Console?

thank you @mark-piller, I understand that there is only one route.

However, I can ignore the HTTP status and just check the payload, and if it contains code 1999 it means the migration is ongoing, while if returns no payload or a different code, then there is no migration.

Otherwise I can ignore the code and payload altogether and just look at the HTTP request return code, which will be always 200 or 503 only the migration is ongoing.

This is what I mean as the two approaches, basically either the code returned by the request, or the code that is part of the payload. Currently we implemented a mechanism to check the code in the payload (so 1999), ignoring the HTTP code, and I am asking if this is fine (and therefore checking one code or the other still provides always the same outcome) or if you foresee cases where we should check for the HTTP code (and in that case I don’t see what is the 1999 code for).

Thank you for any input on this, I hope I’ve explained the scenarios better now.

Yes, what you described is reasonable, as long as the code deals with the scenario when there is no payload.

Regards,
Mark

1 Like

In the export files page, you have a file per export if it does not fail. If it does fail, then there is no file (no zip containing the data) and the log file is updated only during the last successful export as I have mentioned, and this is indicated by the timestamps inside the file, this is the content of the only log file:

07:33:42 EXPORT Export started
07:33:42 EXPORT Exporting table: Users
07:33:44 EXPORT Generating export.zip
07:33:45 EXPORT Uploading export.zip to File Service
07:33:45 EXPORT Export finished

As well as the Timestamp property that I can see in the console (which indeed refers to a time that matches the last zip file that was exported successfully).

Unfortunately when trying to export all the tables data is not exported, and there is no feedback apart from what I have copied earlier (which I receive via email), with also no information in the single log file present.

Hello @Alessandra_Saviotti,

The internal ticket BKNDLSS-20527 created. We’ll investigate this issue as soon as possible.

Regards,
Olha

Thanks, please see my other posts for the application id