Service Incident - December 3, 2024 - Explore | EU - Explore 5xx level errors and access issues

SUMMARY

December 3, 2024 04:24 PM UTC | December 3, 2024 8:24 AM PT
Between 15:09 and 15:37 UTC today, December 3, 2024, Explore customers in EU experienced 5xx level errors and access issues reaching dashboards and reports. These issues have been resolved and access has been restored at this time.

POST-MORTEM

Root Cause Analysis

The incident was caused by a mistake in a technical command during a system update. This error led to a misdirection of traffic, causing issues for our European customers trying to access their Explore dashboards.

Resolution

Our technical team quickly corrected the issue by redirecting the traffic to the appropriate system, which restored access to the Explore dashboards for all affected customers.

Remediation Items

Review Procedures: We will re-evaluate our operational procedures to ensure that all technical settings are correct before making changes.
Improve Technical Tools: Enhancements will be made to our traffic management tools to prevent similar errors in the future.
Reduce Single Points of Failure: We are investigating ways to ensure that our systems are more resilient and do not rely on a single component, improving overall reliability.
Strengthen Review Processes: We will implement more thorough checks for our technical procedures to catch potential issues before they affect our customers.
Enhance Service Reliability: We will review our service performance standards to ensure we consistently meet high expectations for uptime and reliability.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.

SUMMARY

Root Cause Analysis

The incident was caused by a mistake in a technical command during a system update. This error led to a misdirection of traffic, causing issues for our European customers trying to access their Explore dashboards.

Resolution

Our technical team quickly corrected the issue by redirecting the traffic to the appropriate system, which restored access to the Explore dashboards for all affected customers.

Remediation Items

Review Procedures: We will re-evaluate our operational procedures to ensure that all technical settings are correct before making changes.

Improve Technical Tools: Enhancements will be made to our traffic management tools to prevent similar errors in the future.

Reduce Single Points of Failure: We are investigating ways to ensure that our systems are more resilient and do not rely on a single component, improving overall reliability.

Strengthen Review Processes: We will implement more thorough checks for our technical procedures to catch potential issues before they affect our customers.

Enhance Service Reliability: We will review our service performance standards to ensure we consistently meet high expectations for uptime and reliability.

FOR MORE INFORMATION