Service Incident - February 28, 2025 - Help Center | Multiple Pods - Access issues and 429 Errors

SUMMARY

On February 28, 2025 from 11:00 UTC to 12:00 UTC, customers using Help Center in multiple Pods experienced 429 errors and issues accessing Help Center pages.

TIMELINE

February 28, 2025 12:22 PM UTC | February 28, 2025 04:22 AM PT
We're happy to report that the issues causing Help Center to be unavailable, returning 429 errors have now been resolved. Thank you for your patience.

February 28, 2025 12:04 PM UTC | February 28, 2025 04:04 AM PT
We are seeing Help Center services recovering and our engineering team continues to monitor the situation. In case you still receive 429 errors when oprning Help Center, please reach out to us!

February 28, 2025 11:51 AM UTC | February 28, 2025 03:51 AM PT
Our engineering team is currently working on an issue causing Help Center to return a 429 error for our customers across multiple Pods. We will provide further updates as soon as they become available.

POST-MORTEM

Root Cause Analysis

This incident was caused by an infrastructure configuration change. This configuration was unable to handle connections effectively, leading to timeouts and failures when the application attempted to connect. As a result, customers experienced rate-limiting and 429 errors, blocking their requests and access to the Help Center.

Resolution

To resolve this issue, the team rolled back the recent infrastructure changes. This action restored the availability of the service, allowing the application to successfully establish connections and process customer requests.

Remediation Items

1. Implement improved monitoring and alerting for connection errors to identify issues before they affect customers.
2. Revise the existing infrastructure documentation to clarify the limitations of this intended change.
3. Enhance the Help Center’s error handling to provide clearer communication to users experiencing rate-limiting, such as updating the messaging for 429 errors.
4. Conduct a thorough review of the code responsible for handling connections to prevent similar failures in the future.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.