Service Incident - May 21, 2025 - Support (Messaging) | All Pods - Errors loading conversations in tickets

Summary

On May 21, 2025 from 05:13 UTC to 05:51 UTC, Zendesk Support (Messaging) customers on multiple pods experienced issues viewing tickets and communicating with end users in the Messaging interface. Various error codes and error messages prevented agents and end users from communicating with each other.

Timeline

May 21, 2025 05:54 AM UTC | May 20, 2025 10:54 PM PT
We are currently aware of an issue in Support where customers are encountering the error message, "Message couldn’t be loaded." Our engineering team is actively working to resolve the problem. We will provide an update within the next 30 minutes.

May 21, 2025 06:08 AM UTC | May 20, 2025 11:08 PM PT
Our engineering team has resolved the issue. Our engineers immediately performed a rollback to resolve the issue. To ensure the fix takes effect on your end, please perform a hard refresh or clear your cache and cookies. Thank you for your patience and cooperation. A final update will be provided shortly.

May 21, 2025 07:18 AM UTC | May 21, 2025 12:18 AM PT
We are pleased to confirm that the issue affecting Support, which caused some customers to receive the error message “Message couldn’t be loaded,” has been resolved. Thank you for your patience and understanding while our team worked to address this.

Root Cause Analysis

This incident was caused by the introduction of a new field in the API call responsible for fetching ticket conversation data in Support (Messaging). The new code was deployed to accommodate the new field and associated mappings. As part of this release, we unintentionally introduced an invalid mapping of boolean types that resulted in malformed data to be returned by our backend API service. This resulted in the ticket conversation errors experienced by our customers.

Resolution

To fix this issue, our team identified the deploy and rolled it back.

Remediation Items

1. Identify and fix testing gaps that allowed this issue to escape detection in pre-production environments (including smoke tests).
2. Investigate canary phase improvements to surface issues before they become widespread.
3. Investigate and implement explicit backwards compatibility checks during code design phase.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.