"server temporarily unavailable" when trying to access the files on dashboard

app ID D12401BA-C118-2CC9-FF7D-A4B0700D0600

I’m trying to access the list of files, on my app’s dashboard, in order to export the current list of files, but the loading spinner runs without stoping, and I finally get a “server temporarily unavailable” banner at the top of the screen.

Is there any maintenance operation on your side ?

Thanks you for your help,
regards,
loïc b.

Hi loïc,

We are already looking into it.
Could you please also tell us, when did this start happening? Maybe you added some file just before or did any other operation?

Hi, thank you for your message,

It’s hard to say when it’s started. I did not do anything before accessing the dashboard, and go the files tab.
All I can say is that I have this problem for the given app, but not for my other app (the app for the development platform : 30D7FA5B-421F-CD39-FF1D-A77C9FDDEA00). Could it be related to the number of files ? I know I may have a “lot” of files on the root directory…

Yes, I see that there’s a lot of files. It may be the reason, though it should not be.
We’re trying to reproduce this issue and will inform you when we have any results.

Hi again,

The investigation showed that our file system failed to handle this amount of files.
As a workaround, I can move the files into directories, which will make them available from console. The general recommendation is to have no more than 1000 files in a directory.

Hi Sergey,
thank you for your message.

I must say this is quite a constraint. I would have not think that 1000 files would be “too much” for a file system.

Can you please confirm that this problem only impact the console, and not the API itself ? <=> does it impact our published app ?

Moving the files is not possible right now, it would break the mobile application that is published for several months.
We will have to think about a proper script that move the files into a new directory organization, and update the file references (from the database) in same time.
We also have to update the client app, and publish the update, to store the files to come in the new organization.

A new organization would be based on something like year/month/day, but even though, there will be no formal guaranty that none of those folders will ever contain more than 1000 files, so I’m not sure yet.

I will have to list and propose all possible solutions to my company’s ceo, including the possibility of deploying our own server, before taking any action. Will you let me know if any other quick solution shows up on your side ?

Thanks Sergey,
loïc b.

Actually, it depends on the operations you perform. The reason you get a timeout on Backendless Console is that it sorts all the files and folders by default - this is the most time-consuming operation there. We are planning to optimize it and make the sorting optional - this way you will be at least able to view the files without a request timeout.

The same goes to API: the operations which require traversing all the files in the directory, like sorting, counting etc, will be much slower when you have 10k+ files than with 1000 files; but the other operations should still work fine.

I advice you having max 1000 files in a directory, because this case shows the best performance. There are many approaches to the distribution of files, for example the hash directory structure and others. Sure, it’s not a requirement: you may still use your existing approach, but be aware of the limitations described above.

Thank you for your explanation.

I’m having a look to the business logic section right now : implemeting/deploying a timer to be executed once seems to be the most efficient way to write the script I talked about, to move the files and update the database. I’m going to have to deeper look to the documentations, but could you just confirm that the right place to start ?
Thanks Sergey,
l

Sergey,

I’ve took some time to look into the documentations, and I’ve made a test of a simple Business Logic Timer written in Javascript. It seems satisfactory and I should be able to do what I want using this method.
What I plan to do is

  • write a Backendless Timer that retrieve all the existing objects from database that contain a non-updated file path, move the file to a new location in a sub-folder, and update the object file path.
  • test/debug this Timer on my “development” application
  • check that everything is working as expected
  • request your help, if that is something possible, in a few days, to backup the files of the “production” app (I can’t access the console, so I can’t export the directory)
  • deploy the Timer on the production app and execute it.

I will then publish a new version of my iOS app to reflect the change of the file organization, and use the Timer once in while to clean the files (until all iOS clients are updated).

Does that sound OK to you ?

Sergey,

my script is ready, I’ve done some unitary tests that show a good result.
I’m going to make more tests on my development application, with a biggest populated database.
Meanwhile, would you be able to backup the production application (D12401BA-C118-2CC9-FF7D-A4B0700D0600) and send me the download links in a email ? What I mean is a complete export of the database and the files, since I can not do it myself using the Backendless console. ?
Is that’s something you can do, and once it’s done, I will be able to apply the script to the production app.

Thank you a lot for your help.
loïc b.

“would you be able to backup the production application”
Are you talk about full dump of file repository of your app ?

yes, exactly.

Thank you Oleg.

Could you try this:

  • go to the “Files” section;
  • press the rightmost button “ZIP Directory”
  • download archive and check whether all ok.

:slight_smile: Ok, so , the whole story from the beginning is : I can’t access the Files section, I need to deploy a Business logic Timer to update the file organization and the database to fix the problem, and before to do that, I’d rather backup the whole thing. That’s why I’m requesting a bit of help.

thanks
l.

I understood. Thank you for patience.
Usually it is considered as paid support. But we will try to resolve this issue.
Today i will give you the answer.

Thank you Oleg, I’ve waited for your answer but I suppose you had other things to do. I has no choice and try to manage on myself, I’ve written a script to backup manually all the files. Then I deployed the business logic timer to clean the database and the filesystem. The problem seems to be fixed now.

As i mentioned before - it is paid service. Please, contact to sales@backendless.com.
Thank you for understanding.