Service Incident - April 8, 2025 - Zendesk QA | All Pods - Scorecard and review submission failures

Summary

On April 8, 2025 from 12:27 UTC to 16:13 UTC, Zendesk QA customers experienced multiple issues with the product including scorecard loading failures and review submission failures.

Timeline

April 08, 2025 03:02 PM UTC | April 08, 2025 08:02 AM PT
We have received reports of the QA Scorecards failing to load in Zendesk QA on multiple pods. We are currently engaging our engineers to assist with this issue. Investigation is underway. We apologize for this inconvenience and we will be providing the next update in 30 minutes.

April 08, 2025 03:37 PM UTC | April 08, 2025 08:37 AM PT
We continue to investigate the issue of the QA Scorecards failing to load in Zendesk QA on multiple pods. Our engineers have rolled back some potentially problematic changes. We appreciate your patience and we will provide the next update in 30 minutes.

April 08, 2025 04:21 PM UTC | April 08, 2025 09:21 AM PT
We are pleased to inform you that the issue of the QA Scorecards failing to load in Zendesk QA on multiple pods has been resolved. Our engineers confirmed that the fix was successful. We thank you for your patience on this issue.

Root Cause Analysis

This incident was caused by a bug in a recent change to our backend real-time messaging service that manages communication between multiple applications. The issue occurred intermittently when concurrent publishing events were handled.

Resolution

To fix this issue, our team identified the change and rolled our application back to a previous working version. The issue was subsequently fixed prior to re-releasing it back into our product.

Remediation Items

1. Investigate the possibility of adding a recovery handler to our internal service.
2. Additional monitoring to catch similar failures if they happen again.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.