Service Incident - July 5, 2024 - Talk | All Pods - Errors Retrieving Call Recordings

Summary

On July 5, 2024 from 10:54 UTC to 13:36 UTC, we experienced an issue that temporarily prevented some of our Talk customers from accessing their call recordings. This was due to a change in our system that inadvertently caused an error, impacting the availability of these recordings. Once the issue was identified, we reverted the change, restoring normal service.

Timeline

July 05, 2024 01:52 PM UTC | July 05, 2024 06:52 AM PT
Changes were successfully reverted to the previous working state. Logs indicate that our recovery process is functioning and should recover recordings within the next couple of hours. Please allow time for the system to process any previously unavailable recordings. A simple page or browser refresh/update may be required. Thank you for your patience while we worked on this.

July 05, 2024 01:31 PM UTC | July 05, 2024 06:31 AM PT
We are currently aware of and working on reverting recent changes that have impacted Talk customers, causing “Call recordings” to be unavailable across multiple Pods. The next update will be provided in 30 minutes.

POST-MORTEM

Root Cause Analysis

The issue was caused by a change in our system configuration that affected how call recordings were stored. Specifically, a setting was altered that directed our service to an incorrect storage location, resulting in errors when trying to save call recordings.

Resolution

Upon identifying the issue, we quickly reverted the change that caused the problem. This rollback restored the correct configuration, and our service resumed normal operation.

Remediation

To prevent this from happening again, we are taking the following steps:

1. Improved Testing: We will enhance our testing processes to catch similar issues before they reach production.
2. Configuration Management: We will update our system to use separate settings for different services, ensuring that changes to one service do not inadvertently impact another.
3. Monitoring and Alerts: We will improve our monitoring and alerting systems to detect and respond to issues more quickly.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.