Service Incident - September 19, 2024 - All Products | All Pods - Authentication Issues

SUMMARY

On September 19, 2024 from 08:10 UTC to 09:11 UTC, customers using multiple products including Chat, Talk, Support, and Explore across multiple Pods experienced authentication issues, including HTTP 401 and HTTP 500 errors.  

Timeline

September 19, 2024 01:23 PM UTC | September 19, 2024 06:23 AM PT
The authentication issues affecting multiple products and Pods, which specifically caused “This page isn’t working” and HTTP Error 500 messages for customers accessing Zendesk Explore, have been fully resolved. Customers should no longer experience any errors, and normal functionality has been restored. Thank you again for your patience during this time.

September 19, 2024 09:36 AM UTC | September 19, 2024 02:36 AM PT
While our team is actively working to fully resolve the problem and has implemented a potential fix, we are seeing an initial improvement with a decrease in errors. We will continue to monitor the situation and keep you updated with any further developments. Thank you for your patience.

September 19, 2024 09:02 AM UTC | September 19, 2024 02:02 AM PT
We are currently experiencing authentication issues that are affecting multiple products and Pods, including Chat, Talk, Support, and Explore. Our team is actively investigating the problem, and we will provide more information as soon as we have further details. Thank you for your patience and understanding.

POST-MORTEM

Root Cause Analysis

This incident was caused by a misconfiguration on a backup network path, which is only used as a failover when receiving errors from our content provider. The non-standard ports used by this backup network path were improperly configured, causing errors when the backup path was initiated in response to such errors.

Resolution

The issue was resolved when the errors from our content provider subsided, and the network path automatically reverted to the primary. 

Remediation Items

1. Port configurations have been updated to prevent this issue from occurring.
2. We are adding detailed error descriptions and monitors for invalid credentials.
3. We have updated our processes and checks to ensure the port misconfiguration does not reoccur.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.