Service Incident - November 21, 2024 - WFM | All Pods - Delays in Ticket Status updates, Metric updates, and Agent Activities

SUMMARY

On November 21, 2024 from 9:59 UTC to 12:20 UTC, some Zendesk WFM customers across all Pods experienced delays in Ticket Status updates, Metric updates, and Agent Activities.

TIMELINE

November 21, 2024 12:42 PM UTC | November 21, 2024 04:42 AM PT
The delays impacting some Workforce Management customers have been resolved. We apologise for the inconvenience.

November 21, 2024 11:35 AM UTC | November 21, 2024 03:35 AM PT
Our teams continue to work to resolve the delays that some Workforce Management customers are currently experiencing with Ticket Status updates, Metric updates, and Agent Activities. Some customers may see improvements already. We will provide an update as soon as one is available.

November 21, 2024 10:58 AM UTC | November 21, 2024 02:58 AM PT
Some Workforce Management customers are currently experiencing delays in Ticket Status updates, Metric updates, and Agent Activities. We apologise for the inconvenience. Next update in 30 minutes.

POST-MORTEM

Root Cause Analysis

This incident was caused by a misconfiguration related to the introduction of a new user data table in WFM. The new table was intended to standardize the user identification process, but the user syncing job had been on hold. During this time, it incorrectly used the timezone property from the API instead of a more accurate timezone property, resulting in corrupted data.

Resolution

To resolve this issue, we manually updated the invalid timezone entries in the users table to their correct formats, ensuring that all timezone values were valid and consistent with the required format.

Remediation Items

1. Implement validation checks to avoid throwing exceptions on invalid timezone formats in the future.
2. Create alerts for monitoring system lags or performance issues to proactively identify these issues before they escalate.
3. Enhance the user syncing job to ensure it consistently pulls the correct timezone data from the API.
4. Conduct thorough smoke tests on new features before deployment to catch similar issues early in the development cycle.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.