Summary
On September 9, 2024 from 14:16 UTC to 19:40 UTC, a small subset of Zendesk Chat and Support Messaging (Agent Workspace) customers experienced delays and errors in chat and messaging interactions.
Timeline
September 09, 2024 10:06 PM UTC | September 09, 2024 03:06 PM PT
We are investigating reports of errors sending messages in Messaging channels powered via Sunshine Conversations, and our team is investigating. More information will be posted shortly.
September 09, 2024 10:19 PM UTC | September 09, 2024 03:19 PM PT
We have confirmed an issue affecting messaging via social channels powered by Sunshine Conversations, causing some messages to not send to end-users. We will provide additional updates in 30 minutes or when we have new information to share.
September 09, 2024 10:53 PM UTC | September 09, 2024 03:53 PM PT
Our team continues to investigate the issue affecting outgoing messages in multiple Messaging channels powered by Sunshine Conversations. We will post further updates in one hour or as soon as we have a substantive update to share.
September 09, 2024 11:20 PM UTC | September 09, 2024 04:20 PM PT
We are seeing recovery in our logs after a software fix had been deployed, and have successfully tested this as well. Please perform a hard refresh of your browser and let us know if the problem is still existing at your end.
September 09, 2024 11:56 PM UTC | September 09, 2024 04:56 PM PT
We have confirmation that the deployed software fix has solved the issue that was affecting messaging via social channels. Many thanks for your patience as we got to this point.
Root Cause Analysis
This incident was caused by a configuration error committed during a backend database migration. As part of deploying business continuity mechanisms during the upgrade, a vital connection string was misconfigured, preventing our Chat and Messaging applications from communicating with the database.
Resolution
To resolve this issue, we reverted the connection string to allow the applications to access the database correctly and eliminated the unexpected behaviors experienced by customers.
Remediation Items
- Clean up connection strings [Completed]
- Review and improve existing implementation tools [Scheduled]
- Create additional alerts to monitor for similar misconfigurations in the future [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
0 comments
Article is closed for comments.