Service Incident - August 15, 2024 - Chat | All Pods - Chat triggers being fired unexpectedly

SUMMARY

On August 15, 2024 from 09:45 UTC to 13:19 UTC, Chat customers on multiple pods may have experienced Chat triggers firing incorrectly.  This affected chat tagging, routing, and how automated messages were sent, and customers may have experienced unexpected Chat behaviour.

Timeline

August 15, 2024 01:06 PM UTC | August 15, 2024 06:06 AM PT
We’re currently investigating reports of Chat triggers incorrectly firing, across all Pods. More details soon.

August 15, 2024 01:18 PM UTC | August 15, 2024 06:18 AM PT
We are rolling back a recent deploy related to Chat triggers. Please note that a possible side effect of this incident is that incorrect Chat tags can be added to chats, and this can cause misrouting. Next update in 30 min.

August 15, 2024 01:29 PM UTC | August 15, 2024 06:29 AM PT
The rollback for the issue related to Chat triggers firing incorrectly has finished updating across all pods. We appreciate your patience as we monitor the situation until full resolution.

August 15, 2024 01:46 PM UTC | August 15, 2024 06:46 AM PT
Thank you for your patience while we worked through this incident. Chat triggers are now working as expected, and we no longer see errors related to this issue. We are marking this as fully resolved.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by a new mechanism to evaluate Chat trigger conditions that skipped certain scenarios, resulting in a default action of firing off trigger actions. 

Resolution

To fix this issue, a new software version that addresses the trigger condition evaluation gap was deployed.  After further monitoring, we received confirmation of recovery.

Remediation Items

1. Adding anomaly detection for unusual volumes of trigger actions [Scheduled]
2. Implementing additional tests to cover more complex condition scenarios [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.