Hello,
whats the best way to load 50,000 rows of data from backendless, i’ve read the topics on loading data efficently using async, but when i employ them, i only get 9 or 10 pages of data loaded into memory before i get a “bad gateway” error from the server… i am not sure where i could be going wrong, so any input is greatly appreciated…
Thanks.
Hi James,
Are you talking about retrieving data from Data Service to your application? The approach with loading them page by page should be fine, no matter sync or async. Can you please share the code you use to accomplish this?
Below is a snippet of the code i’m using, this is the sync version of the code, it seems to load 10 - 15 pages then throws an internal server error “Bad gateway”
protected static IDataStore<Log> logDataStore;
private static List<Log> _allEntries = new List<Log>();
protected void Page_Load(object sender, EventArgs e)
{
logDataStore = Backendless.Persistence.Of<Log>();
BackendlessDataQuery bdq = new BackendlessDataQuery();
bdq.PageSize = 100;
QueryOptions qo = new QueryOptions();
qo.PageSize = 100;
List<string> sortBy = new List<string>();
sortBy.Add(“Date_Time ASC”);
qo.SortBy = sortBy;
bdq.QueryOptions = qo;
BackendlessCollection<Log> allEntries = new BackendlessCollection<Log>();
if (logDataStore != null)
{
allEntries = logDataStore.Find(bdq);
}
while (allEntries.Data.Count > 0 && _allEntries.Count < 64000)
{
_allEntries.AddRange(allEntries.Data);
allEntries = allEntries.NextPage();
}
datagrid.datasource = _allEntries;
datagrid.databind();
}
I have created an internal task BKNDLSS-13285 to investigate large data loading. We shall notify you here as soon as we have any results.
Hi James,
Out of curiosity, what if you put a slight delay right before this line:
allEntries = allEntries.NextPage();
Would you still get an error that way?
Regards,
Mark
Mark,
i’ve tried adding delays befdore and after the .NextPage call and i still get the Bad gateway error.
James, could you please let me know your app ID, I’d like to see if I get the same error when loading data from your app.