Service Incident - August 7, 2024 - Support | All Pods - Ticket views issues

SUMMARY

On August 07, 2024 from 20:48 UTC to 23:27 UTC, Support customers across all Pods experienced issues with ticket views, including solved tickets not disappearing, slow updates, and incorrect numbers displaying.

Timeline

August 07, 2024 10:00 PM UTC | August 07, 2024 03:00 PM PT
We are investigating reports of Support ticket views not updating or showing incorrect tickets. Next update in 30 minutes or when we have new info to share.

August 07, 2024 10:08 PM UTC | August 07, 2024 03:08 PM PT
Our engineers have made a change and we are now seeing performance improvements across all pods. Next update in 30 minutes or when we have new info to share.

August 07, 2024 10:24 PM UTC | August 07, 2024 03:24 PM PT
Performance continues to remain stable across all pods and we will continue to monitor performance for the next hour. Next update in one hour.

August 07, 2024 11:29 PM UTC | August 07, 2024 04:29 PM PT
After continued monitoring, we are seeing stability, and have confirmation that the issue impacting Support ticket views not being updated has been resolved. Thank you for your patience as we got to this point.

POST-MORTEM

Root Cause Analysis

This incident was caused by a process breakdown during a blue/green cluster switch, specifically due to an incomplete deployment that led to configuration errors in the remaining production Pods. 

Resolution

To fix this issue, the engineering team manually scaled up the indexers on the active cluster to process accumulated ticket updates. They then resumed the cluster-switch procedure with the correct configuration.

Remediation Items

1. Review of the process used for cluster switch is needed to ensure it includes thorough post-deployment checks to prevent incomplete deployments from going unnoticed.
2. Develop automation to handle the cluster-switch process, reducing the need for manual intervention and minimizing the chances of human error.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.