Service Incident - March 28, 2025 - Support | Automation jobs not running on some tickets on Pods 13, 15, 26, and 30

Summary

From March 27, 2025 at 09:00 UTC until March 28, 2025 at 18:00 UTC, Zendesk Support customers on Pods 13, 15, 26, and 30 may have experienced issues with automations not running on some tickets. This issue required customers to log out and log back in to restore normal operations. This incident affected all active accounts within the last 14 days on these pods, leading to a noticeable decrease in automation jobs.

Timeline

March 28, 2025 09:20 PM UTC | March 28, 2025 02:20 PM PT
Between March 27, 2025 at 8:00 UTC and March 28, 2025 at 18:00 UTC, automation jobs across multiple pods experienced issues firing and some automations may have not fired on some tickets. Please log off and log back into your account to ensure automations run again at their next hourly run time. We apologize for any disruption this may have caused.

Root Cause Analysis

This incident was caused by a code change in a backend service for Zendesk Support that stopped updating the internal job for accounts which resulted in the internal automation job failing to run. While the change aimed to enhance performance, it unintentionally disrupted automation jobs.

Resolution

To fix this issue, our team identified the problematic code and rolled it back to a previous working version.

Remediation Items

1. Review and adopt alternative solutions for internal Automations jobs.
2. Additional monitoring and alerting to ensure we are notified when automations stop running.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.