Service Incident - April 29, 2025 - Pod 17|Support - Performance issues on POD17

SUMMARY

On April 29, 2025 from 10:20 UTC to 10:42 UTC, customers in Pod 17 experienced severe delays and performance issues when accessing their Zendesk applications and services. 

Timeline

April 29, 2025 11:38 AM UTC | April 29, 2025 04:38 AM PT

We received reports from customers on Pod 17 experiencing slowness with Zendesk products between 10:20 and 10:40 UTC. Our engineers have confirmed that the system has recovered and are currently investigating the root cause. We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. We will provide updates as they become available.

April 29, 2025 01:45 PM UTC | April 29, 2025 06:45 AM PT

We continue to investigate the cause of the degradation of application response time for Zendesk customers in Pod 17. Our engineers are currently identifying the root cause and engaging our partners for their investigation. We appreciate your patience and understanding as we work to provide you with an accurate resolution for the root cause of this issue. We will provide updates as it becomes available.

April 29, 2025 03:14 PM UTC | April 29, 2025 08:14 AM PT

We are pleased to inform you that the degradation of application response time for Zendesk customers in pod 17 has been resolved. We thank you for your understanding and patience as we worked to resolve this issue.

Root Cause Analysis

This incident was caused by the lack of sufficient capacity allocation for the service during a proxy deployment, which resulted in a severe performance degradation and delayed response for customers in Pod 17 when accessing their Zendesk applications.

Resolution

To fix this issue, the engineers allowed the ongoing internet-facing proxy deployment to finish and to allow the capacity allocation to revert back to normal levels.

Remediation Items

1. Increase the allocated capacity by adding more proxy nodes to handle the surge of requests during peak hours (when the issue occurred).
2. Setup additional monitoring for proxy node activity to preemptively take action and avoid further performance issues.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.