Service Incident - September 11, 2024 - Guide | All Pods - 429 errors when submitting tickets in Help Center

SUMMARY

On September 11, 2024, from 09:07 UTC to 10:13 UTC, end-users and customers using Guide experienced difficulties submitting inquiries and requests due to a 429 error message when trying to create tickets. This error, which stated “Rate limited by Zendesk edge protection”,“Status”:“429", affected end-users submitting tickets via the Help Center or web form. 

Timeline

September 11, 2024 09:19 AM UTC | September 11, 2024 02:19 AM PT

We’re currently investigating customer reports of 429 errors when accessing Support or submitting tickets via Help Center in Guide. More information to come.

September 11, 2024 09:26 AM UTC | September 11, 2024 02:26 AM PT

We have rolled back a recent update that was causing 429 errors across all Pods. Please make sure to clear your cache and cookies to ensure you have the most updated version of the system available. Thank you for your ongoing patience.

September 11, 2024 09:57 AM UTC | September 11, 2024 02:57 AM PT

We have received confirmations that customers are no longer experiencing 429 errors when trying to submit tickets via different methods in Zendesk. We are marking this as fully resolved now.

POST-MORTEM

Root Cause Analysis

This incident was caused by the migration of rate limit rules to a new ruleset in the content delivery network (CDN) system, which did not function as expected. The new rules, although tested, led to excessive rate limiting, resulting in a significant number of customers receiving 429 error messages when attempting to create support tickets. 

Resolution

To fix this issue, we reverted the change that had migrated the rate limits. This action quickly mitigated the impact, rapidly restoring normal functionality for customers.

Remediation Items

1. Implement a dedicated monitoring system for each role change in the CDN to catch issues early.
2. Enhance existing implementation tools to ensure smoother transitions during rule migrations.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.