Service Incident - March 10, 2025 - Messaging via Sunshine Conversations | Pod 26 - Delays and errors sending messages

SUMMARY

On March 10, 2025 from 18:41 UTC to 19:06 UTC, customers using Messaging via Sunshine Conversations on Pod 26, as well as our support.zendesk.com instance, experienced delays and errors sending messages.

TIMELINE

March 10, 2025 07:09 PM UTC | March 10, 2025 12:09 PM PT
We have found that some connections to Sunshine Conversations SDKs may have experienced continued errors until approximately 19:06 UTC; however, that issue is now also resolved.

March 10, 2025 07:03 PM UTC | March 10, 2025 12:03 PM PT
Between 18:42 and 18:53 UTC on March 10, 2025, Messaging via Sunshine Conversations on Pod 26 experienced delays and errors sending messages. This included messages to contact Zendesk Support via our widget at support.zendesk.com. The degradation has ended and messages should send as expected at this time. We apologize for any disruption this may have caused.

POST-MORTEM

Root Cause Analysis
This incident was caused by a misconfiguration of a database secret for Sunshine Conversations, which resulted in the workloads in Pod 26 being unable to authenticate to the databases. The connection string lacked essential configuration parameters, preventing successful connections to the database clusters.

Resolution

To resolve this issue, the team rolled back the secret change as soon as the problem was identified, canceled the ongoing deployment, and redeployed the previous secret configuration that allowed for proper database connectivity.

Remediation Items

1. Investigate configuration to prevent traffic from being sent to faulty new deployments.
2. Add a boot check for database connectivity and authentication to ensure that services can connect to the database before proceeding with deployment.
3. Add a validation step for the database secrets to ensure correct configuration before deployment.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.