Spike in ECONNRESET Errors

We have observed a significant spike in ECONNRESET errors daily starting from month of December. Our monitoring confirms we are well below our concurrent request limits and have not increased our traffic volume.


Symptoms:

  1. Connections are being abruptly closed by the server.
  2. Occurs on standard endpoints that previously performed reliably.
  3. No correlation with high-load periods on our side.

Constraints:

Our architecture relies on connection stability. We do not use client-side retry logic for these methods and cannot implement it as a patch for server-side instability.

Has there been a recent change to the load balancer timeouts or keep-alive configurations on the Backendless side? We suspect a regression or infrastructure change on your end.

Please investigate the server logs for our App ID: 4A47197B-AE30-FA84-FF56-0071F4010900

Hello @Francis_Bagas

Thank you for providing the detailed information about the ECONNRESET errors you’re experiencing. We understand how critical connection stability is for your architecture, and we want to assure you that we’re taking this issue seriously.

I’ve conducted an initial investigation of your application logs (App ID: 4A47197B-AE30-FA84-FF56-0071F4010900) covering the past three months. While I haven’t yet been able to identify a pattern matching the spike you’re observing since December, I want to work with you to get to the root cause of this issue.

To help us perform a more thorough analysis and correlate our logs with your observations, could you please provide the following additional information:

  1. Timezone Information: What timezone are the timestamps in your screenshots? This will help us accurately match the error occurrences in our logs.
  2. Service/Method Details: Which specific Backendless service or API endpoints are experiencing these errors? From the screenshots, I can see various method names (QC_Timer_CS_Generic, beforeUpdate, ZoomNotificationsLateStudentAndTutor, etc.), but knowing the exact service context would be helpful.
  3. Log Export: If possible, could you provide the error logs in a text format (CSV, JSON, or plain text) rather than screenshots? This would allow us to perform more detailed pattern analysis and correlation.
  4. Recent Changes: Have you made any changes to your application code or infrastructure around the time the issues started?

Regarding your question about infrastructure changes: To the best of my knowledge, there haven’t been any changes to load balancer timeouts or keep-alive configurations on our side that would cause this behavior. However, I will double-check with our team to ensure nothing was modified that could potentially impact your connections.

We genuinely want to resolve this issue for you as quickly as possible. The additional information will help us narrow down the investigation and identify whether this is related to a specific service, network path, or timing pattern.

Please feel free to share any other observations or patterns you’ve noticed. We’ll continue monitoring your application closely and will update you as soon as we have more information.

Regards,
Volodymyr

Hello @Volodymyr_Ialovyi,

Thank you for your response.
Here is the additional information:

  1. The timezone used in the screenshots is UTC+8, but the list below shows the services/methods in UTC.
  2. Here is the data from our errors table, where we log the errors we encounter.
[
  {
    "code": "\"ECONNRESET\"",
    "method": "UpdateChargebeeWithCloseByLead",
    "___class": "errors",
    "time": 1763293887927,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1764593786585,
    "message": "\"socket hang up\""
  },
  {
    "code": "ECONNRESET",
    "method": "DatabaseAddNextMSCalendarSessionObject",
    "___class": "errors",
    "time": 1761873244575,
    "message": "socket hang up"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "weeklySlotsUpdate_v2",
    "___class": "errors",
    "time": 1767646744810,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761450076550,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761833480228,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseUpdateMSMeetingMSDayOff",
    "___class": "errors",
    "time": 1762207075745,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Finance_Subscription_Changes_to_Desc_Prepaid",
    "___class": "errors",
    "time": 1765284575104,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "usersUpdate",
    "___class": "errors",
    "time": 1761899520836,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterCreate ms_calendar",
    "___class": "errors",
    "time": 1764963820261,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "QC_Admin_Checker",
    "___class": "errors",
    "time": 1762398977407,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "GoogleDriveRefreshCode_v2",
    "___class": "errors",
    "time": 1762628063573,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "ECONNRESET",
    "method": "weekly slots add timer",
    "___class": "errors",
    "time": 1762674852682,
    "message": "read ECONNRESET"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ChargebeeSync",
    "___class": "errors",
    "time": 1761899430794,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseUpdateMSMeetingMSDayOff",
    "___class": "errors",
    "time": 1762619810844,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1762619810827,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Finance_Subscription_Changes_to_Desc_Prepaid",
    "___class": "errors",
    "time": 1768291591551,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseUpdateMSMeetingMSDayOff",
    "___class": "errors",
    "time": 1762658321366,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "ECONNRESET",
    "method": "weekly slots add timer",
    "___class": "errors",
    "time": 1762781065017,
    "message": "socket hang up"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1766489148573,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "ECONNRESET",
    "method": "DatabaseAddNextMSCalendarSessionObject",
    "___class": "errors",
    "time": 1762398825668,
    "message": "read ECONNRESET"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Finance_Subscription_Changes_to_Desc_Prepaid",
    "___class": "errors",
    "time": 1768291760399,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1761873244834,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ChargebeeSyncRefactored",
    "___class": "errors",
    "time": 1761450076605,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "GoogleDriveRefreshCode_v2",
    "___class": "errors",
    "time": 1762658321371,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LastSessionRefresh_v2",
    "___class": "errors",
    "time": 1762180904642,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Finance_Subscription_Changes_to_Desc_Prepaid",
    "___class": "errors",
    "time": 1762624293785,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "AutoRev_Check_Update_Gsheet",
    "___class": "errors",
    "time": 1761873380719,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "GoogleDriveRefreshCode_v2",
    "___class": "errors",
    "time": 1762206861022,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddNextCalendarSessionObject",
    "___class": "errors",
    "time": 1762982477920,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761752656721,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LeadUpdateEvent",
    "___class": "errors",
    "time": 1762398825887,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "SendMissedSessionRecording",
    "___class": "errors",
    "time": 1762674852941,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Reccuring_Email_Notifications_v3",
    "___class": "errors",
    "time": 1766489148584,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "StripeSync",
    "___class": "errors",
    "time": 1761450076637,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761450076556,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LastSessionRefresh_v2",
    "___class": "errors",
    "time": 1762619810787,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ChargebeeSyncRefactored",
    "___class": "errors",
    "time": 1768347415368,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddTutorFromClose",
    "___class": "errors",
    "time": 1766064645592,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LeadUpdateEvent",
    "___class": "errors",
    "time": 1762398825864,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761450077112,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ChargebeeSyncRefactored",
    "___class": "errors",
    "time": 1768558835658,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "StripeSync",
    "___class": "errors",
    "time": 1766064662997,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1764352921851,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseUpdateMSMeetingMSDayOff",
    "___class": "errors",
    "time": 1762398977476,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "weeklySlotsUpdate_v2",
    "___class": "errors",
    "time": 1762628063583,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ZoomTokenRefresh",
    "___class": "errors",
    "time": 1768398266756,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LeadUpdateEvent",
    "___class": "errors",
    "time": 1762624366431,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761450076634,
    "message": "\"socket hang up\""
  },
  {
    "code": "ECONNRESET",
    "method": "weekly slots add timer",
    "___class": "errors",
    "time": 1768398266539,
    "message": "read ECONNRESET"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1762398825921,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1763419616410,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Smartview_Refresh_v2",
    "___class": "errors",
    "time": 1762398977412,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ZoomNotificationsLateStudentAndTutor",
    "___class": "errors",
    "time": 1768460434700,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761752656703,
    "message": "\"socket hang up\""
  },
  {
    "code": "ECONNRESET",
    "method": "DatabaseAddNextMSCalendarSessionObject",
    "___class": "errors",
    "time": 1763453290501,
    "message": "socket hang up"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1767755641604,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddNextCalendarSessionObject",
    "___class": "errors",
    "time": 1764352921885,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Smartview_Refresh Timer",
    "___class": "errors",
    "time": 1761873380572,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1762856524301,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ZoomTokenRefresh",
    "___class": "errors",
    "time": 1768475310551,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "ChargebeeSyncRefactored",
    "___class": "errors",
    "time": 1761833480319,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterCreate ms_calendar",
    "___class": "errors",
    "time": 1762647610645,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1761873380696,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1762619691286,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "SendMissedSessionRecording",
    "___class": "errors",
    "time": 1761752656731,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "CloseUpdateAvailableUnitByChargebeeSubs",
    "___class": "errors",
    "time": 1762432337849,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Reccuring_Email_Notifications_v3",
    "___class": "errors",
    "time": 1763855720926,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Offboarding_All_Contacts_Main_v2",
    "___class": "errors",
    "time": 1768398266760,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddNextCalendarSessionObject",
    "___class": "errors",
    "time": 1762398977486,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LastSessionRefresh_v2",
    "___class": "errors",
    "time": 1762647375714,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "weeklySlotsUpdate_v2",
    "___class": "errors",
    "time": 1767755641620,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1764593786315,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LastSessionRefresh_v2",
    "___class": "errors",
    "time": 1762056408466,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761833480240,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761752656691,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LeadUpdateEvent",
    "___class": "errors",
    "time": 1762690505234,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1761873244819,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "weeklySlotsUpdate_v2",
    "___class": "errors",
    "time": 1765738923027,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "LeadUpdateEvent",
    "___class": "errors",
    "time": 1762875312278,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "CloseUpdateAvailableUnitByChargebeeSubs",
    "___class": "errors",
    "time": 1762398745679,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddMatchingHiringTutorFromClose",
    "___class": "errors",
    "time": 1762658321429,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761833480196,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "UpdateChargebeeWithCloseByLead",
    "___class": "errors",
    "time": 1768291760619,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseAddMissingCalendarObject",
    "___class": "errors",
    "time": 1762624366364,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1762690505247,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1764746694173,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1762628063513,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1763419927518,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "afterUpdate weekly_slots",
    "___class": "errors",
    "time": 1761752656706,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1765284575390,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "ECONNRESET",
    "method": "weekly slots add timer",
    "___class": "errors",
    "time": 1762206860708,
    "message": "read ECONNRESET"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "DatabaseDeleteAutoRevObject",
    "___class": "errors",
    "time": 1762056491983,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "HomeworkSend",
    "___class": "errors",
    "time": 1762658321372,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "GoogleDriveRefreshCode_v2",
    "___class": "errors",
    "time": 1762207075731,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "Finance_Subscription_Changes_to_Desc_Prepaid",
    "___class": "errors",
    "time": 1765972129180,
    "message": "\"read ECONNRESET\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "beforeUpdate weekly_slots",
    "___class": "errors",
    "time": 1763453395752,
    "message": "\"socket hang up\""
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1765702204493,
    "message": "\"socket hang up\""
  },
  {
    "code": "ECONNRESET",
    "method": "DatabaseAddNextMSCalendarSessionObject",
    "___class": "errors",
    "time": 1768398180128,
    "message": "socket hang up"
  },
  {
    "code": "\"ECONNRESET\"",
    "method": "chargebee_payments",
    "___class": "errors",
    "time": 1762982589021,
    "message": "\"socket hang up\""
  }
]

I also noticed another issue: I couldn’t drag the blocks in Codeless and UI Builder.

Regards,
Francis

Hi @Francis_Bagas
I have shared the information about the issues with my colleagues. We will get back to you shortly with an update.

Regarding the drag-and-drop of blocks, I tried to reproduce the issue, but everything works as expected on my side. Could you please let us know which browser and operating system you are using, and in which application this occurred?
Have you also tried using a different browser or device to reproduce the same issue?

Regards,
Sergey