SUMMARY
On March 20, 2025 from 14:32 UTC to March 22 at 16:52 UTC, customers across multiple Pods first experienced 5xx errors, green screens, and access issues. This was followed by a period of some web widget and SunCo traffic encountering intermittent errors.
TIMELINE
March 28, 2025 11:00 PM UTC | March 28, 2025 4:00 PM PT
We are happy to report that our continued monitoring has encountered no further issues or concerns and as such we are fully resolving this incident. Thank you for your patience as our team worked to address and then monitor the widget connection errors. We apologize for any disruption this may have caused.
March 24, 2025 5:44 PM UTC | March 24, 2025 10:44 AM PT
After continuing our work and monitoring through the weekend we are confident that the widget connection issues have been resolved, and no further delays or errors will arise. In the interest of caution, we're going to keep this incident in monitoring status for the time being as we continue to ensure sustained stability. We apologize for the disruption this may have caused, and thank you for your patience during our investigation.
March 21, 2025 11:15 PM UTC | March 21, 2025 4:15 PM PT
As we head into the end of the week, widget connection issues have drastically reduced, but some lingering issues persist. Our work to restore full functionality will continue over the weekend, and we will provide further updates on the status of our investigation when business hours resume at the start of next week. We greatly appreciate your continued patience.
March 21, 2025 8:26 PM UTC | March 21, 2025 1:26 PM PT
While our team continues to work to fully resolve the intermittent widget loading errors affecting a subset of customers, we wanted to clarify that this issue is unrelated to the recently discovered incoming and outbound Sunshine Conversations messaging delays. We thank you for your patience as we work to resolve both issues.
March 21, 2025 5:00 PM UTC | March 21, 2025 10:00 AM PT
Our engineering continues to work toward a solution for the intermittent delays that a subset of customers may experience when trying to connect to web widgets. Most attempted connections should operate as expected; however, If you experience delays or disconnects, the system will automatically reconnect and bring up the widget. We understand how frustrating this is and are working hard to restore full service. We will provide additional updates when we have made progress and have new information to share.
March 21, 2025 10:29 AM UTC | March 21, 2025 03:29 AM PT
Our engineering team continues their work at the highest priority to resolve the remaining SunCo messaging and SDK issues. We will keep you informed with updates as they become available.
March 21, 2025 5:35 AM UTC | March 20, 2025 10:35 PM PT
Our team continues to work on a solution to force the Zendesk Web Widget to reload and load the updated and fixed code we have rolled out. If you are communicating through other channels with an end user experiencing messaging issues through your account's widget, we recommend that they perform a hard refresh of their browsers, which should resolve the widget issue by receiving our fixed code immediately.
March 21, 2025 1:41 AM UTC | March 20, 2025 6:41 PM PT
We would like to clarify that while we mentioned Pod 27 as the primary location of the SunCo messaging impact, it’s important to note that SunCo accounts may not be located on the same pod as your Support instances. As a result, customers and messaging end users across multiple pods may still experience issues if their SunCo account is on Pod 27.
If you are communicating with an end user through non-messaging channels, we recommend that they perform a hard refresh of their browsers, which should resolve the widget issue. We're working on some potential mechanisms to force the widgets to reload and load the updated and fixed code we have rolled out.
Thanks for your continued patience.
March 21, 2025 12:22 AM UTC | March 20, 2025 5:22 PM PT
Our team continues to work at the highest priority to resolve the remaining SunCo messaging and SDK issues. Currently, the most significant impact is occurring on Pod 27, where we are experiencing continued messaging session disconnects for some end users. While the impact on other pods should be minimal, we encourage you to reach out if you are facing any issues on different pods. Please provide details about your end user experience by contacting us using the following contact methods: https://support.zendesk.com/hc/en-us/articles/4408843597850-Contacting-Zendesk-Customer-Support. We appreciate your patience and understanding during this time. We will provide our next update as soon as we have more information to share.
March 20, 2025 9:44 PM UTC | March 20, 2025 2:44 PM PT
Some lingering issues related to SunCo messaging and SDKs persist, and our engineering team is still working to address them and ensure our systems have fully recovered. We will provide further updates when we have new information to share.
March 20, 2025 7:25 PM UTC | March 20, 2025 12:25 PM PT
While the majority of affected functionality has recovered, we are still working to address some lingering symptoms following the 5xx errors and access issues seen earlier today. At this time, some web widget and SunCo traffic may still be encountering intermittent errors. We will provide additional updates in the next two hours or when we have new information to share.
March 20, 2025 5:25 PM UTC | March 20, 2025 10:25 AM PT
We are seeing additional services restored to full functionality and continue to move in a positive direction toward full resolution. Our team will extend their monitoring to ensure that no further errors or access issues are encountered.
March 20, 2025 4:46 PM UTC | March 20, 2025 9:46 AM PT
Our team continues to work on restoring traffic to affected services in a controlled manner to ensure no further issues or degradation returns. Some intermittent errors may be seen attempting to connect to widgets or using SDKs for messaging, and we will post further updates as we move to fully resolve the issues seen today.
March 20, 2025 4:15 PM UTC | March 20, 2025 9:15 AM PT
While we continue to monitor our recovery, we have noted some services which are restarting more slowly than others. Some intermittent errors may be expected as we work to fully resolve the issue.
March 20, 2025 4:04 PM UTC | March 20, 2025 9:04 AM PT
We are beginning to see recovery in the access issues and 5xx errors across multiple pods, and Zendesk Support is once again able to respond to inquiries at support.zendesk.com. We will continue to monitor the situation to ensure full resolution.
March 20, 2025 3:50 PM UTC | March 20, 2025 8:50 AM PT
We have confirmed an issue affecting multiple pods causing 5xx errors, green screens, and access issues. This impact includes Zendesk Support at support.zendesk.com and is affecting our ability to respond to inquiries. Our team is investigating and we will provide additional information soon.
March 20, 2025 3:43 PM UTC | March 20, 2025 8:43 AM PT
We are receiving reports of access issues across multiple pods. Our team is investigating and we will provide further updates shortly.
POST-MORTEM
Root Cause Analysis
This incident was caused by a bug in the Web Widget that led to an infinite loop in the websockets connection handling. When connections failed, the widget continuously attempted to establish new connections without properly closing the previous ones, resulting in an exponential increase in connection attempts.
Resolution
To resolve the issue, the team rolled back the Web Widget to the previous stable version. Additionally, a rate-limiting rule was implemented to manage the excessive traffic and stabilize the service.
Remediation Items
- Review and Improve the Web Widget Rollback Process: Ensure that the rollback process is well understood and effectively prevents bad versions from being deployed.
- Implement Comprehensive Testing: Increase the frequency and scope of smoke tests for the Web Widget to catch similar issues before they reach production.
- Enhance Rate Limiting: Develop and enforce stricter rate limits on websocket connections to prevent overwhelming the infrastructure during peak traffic.
- Introduce a Kill Switch: Implement a mechanism to disable the Web Widget remotely in case of severe issues, allowing for quicker recovery.
- Align Deployment Processes: Ensure that the Web Widget deployment process follows standard multi-phase approaches to minimize risk during releases.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
0 comments
Article is closed for comments.