Summary
On November 7, 2024 from 12:13 UTC to 12:53 UTC, some Zendesk Support customers using Messaging on Pod 28 may have experienced issues creating tickets, loading live conversations and delayed transcription updates.
Timeline
November 07, 2024 01:03 PM UTC | November 07, 2024 05:03 AM PT
We’re experiencing some delays with ticket creation and conversation loading times for messaging customers on pod 28. Our team is working to resolve the issue and restore full service as quickly as possible.
November 07, 2024 01:33 PM UTC | November 07, 2024 05:33 AM PT
We’re pleased to inform you that the issue causing delays in ticket creation and conversation loading times on Pod 28 has been resolved. Thank you for your patience.
Root Cause Analysis
This incident was caused by a bug in a recent code release that sought to resolve an unrelated issue. The issue centred on a query that wasn’t wrapped in a required transaction, resulting in a pile-up of queries leading to the service instability experienced by customers. While the change went through Zendesk’s standard deployment checks and testing regime, this transaction issue was not detected due to insufficient loads in our testing environment.
Resolution
To fix this issue, our team identified the erroneous change and rolled it back.
Remediation Items
- Review and change alerting thresholds to improve incident response time [Completed]
- Review documentation and re-communicate SQL wrapper guidelines [Completed]
- Investigate additional load testing of Messaging features for similar future changes [Scheduled]
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.
0 comments
Article is closed for comments.