Service Incident - October 16, 2024 - Support | Pod 29 - Inbound email delays

Summary

On October 16, 2024 from 06:24 UTC to 14:52 UTC, some Zendesk customers on Pod 29 experienced inbound email processing delays leading to delayed ticket creation and ticket updates in Support.

Timeline

October 16, 2024 12:35 PM UTC | October 16, 2024 05:35 AM PT
Our teams are investigating a delay in email processing which is impacting Zendesk accounts on pod29. We will provide more information as it is available.

October 16, 2024 01:27 PM UTC | October 16, 2024 06:27 AM PT
We are actively working to resolve the service incident that is causing delays in email functionality. We will provide further updates as more information becomes available.

October 16, 2024 01:45 PM UTC | October 16, 2024 06:45 AM PT
We have observed some improvements in our logs concerning the service disruption affecting email delivery. After further investigation we believe these delays only impacted inbound emails. We are still investigating the issue and will provide an update as soon as we have confirmation.

October 16, 2024 02:06 PM UTC | October 16, 2024 07:06 AM PT
We are happy to report that email delays for inbound emails on Pod29 is now solved. Thank you for your patience. We will publish a post-mortem on these events as soon as it is completed.

Root Cause Analysis

This incident was primarily caused by a memory capacity issue in our backend search services. The email delays were exacerbated by a secondary issue introduced by our initial fix that sought to resolve the root cause in the search service. This secondary issue halted email processing for 30-60 minutes before our team fully identified the issues.

Resolution

To fix this issue, our team identified two changes that caused the primary and secondary issues in this incident and rolled back both changes to previous working version of our software.

Remediation Items

1. Increase sensitivity of existing monitors for quicker alerting [Scheduled]
2. Set up additional monitoring for search service [Scheduled]
3. Rebalance search service resources to account for new features [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.