Service Incident - April 16, 2025 - Pod25 | Support and Chat - Customers may experience latency and failures when processing tickets, chats and messaging

SUMMARY

On April 16, 2025 from 04:55 UTC to 05:15 UTC, Zendesk Support and Chat customers on Pod 25 may have experienced multiple issues including ticket creation/update errors, chat connection errors, ticket SLA breaches and issues accepting voice calls.

Timeline

April 16, 2025 05:29 AM UTC | April 15, 2025 10:29 PM PT

We have detected issues on Pod 25 across multiple products since 04:55 UTC. Recovery is being observed, however our team continues to investigate. Some symptoms may include latency and failures experienced by customers when processing tickets, chats, voice, and messaging. More to come.

April 16, 2025 06:01 AM UTC | April 15, 2025 11:01 PM PT

We continue to monitor our systems following full recovery of the performance issues impacting multiple products on Pod 25. Please let us know if you continue to experience errors in our products. We will provide a final update soon.

April 16, 2025 06:01 AM UTC | April 15, 2025 11:01 PM PT

We continue to monitor our systems following full recovery of the performance issues impacting multiple products on Pod 25. Please let us know if you continue to experience errors in our products. We will provide a final update soon.

Root Cause Analysis

This incident was caused by a bug in a third-party container management service (Kubernetes) that led to an unexpected shutdown of an internal messaging broker. This in turn caused a downstream capacity issue that resulted in the various issues experienced by our customers.

Resolution

The cluster handling the internal messaging broker recovered on its own once it went back online, with no intervention.

Remediation Items

1. Review alerting mechanism for improvements to prevent peak time maintenance activities.
2. Investigate customer ticket trend detection for this incident to improve detection time.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.