Service Incident - August 1, 2024 - Support Integrations | All Pods - Loading Apps issue

SUMMARY

On August 1, 2024, from 14:04 UTC to 15:53 UTC, users experienced various issues with Zendesk Apps, such as missing features, loading errors like "We can't load this page right now," and disappearing integrations with the message "There are no apps visible from this location," even though apps were installed.

Timeline

August 01, 2024 03:06 PM UTC | August 01, 2024 08:06 AM PT
We are currently investigating an issue where customers across multiple Pods are unable to load Apps. More information will be provided as it becomes available.

August 01, 2024 03:23 PM UTC | August 01, 2024 08:23 AM PT
We continue our investigation into an issue that is causing customers being unable to load Apps in Support. We appreciate your patience and will provide another update in 30 mins.

August 01, 2024 03:54 PM UTC | August 01, 2024 08:54 AM PT
We are continuing our investigation into the root cause of customers unable to load Apps across multiple Pods. We will provide another update in 60 mins or when more information becomes available. We apologise for any inconveniences caused.

August 01, 2024 04:04 PM UTC | August 01, 2024 09:04 AM PT
We have deployed a fix and customers are now able to see the Support Apps as normal again. Please make sure to refresh your browser. We will now consider this incident resolved, thank you.

POST-MORTEM

Root Cause Analysis

The incident was caused by the revocation of Identity and Access Management (IAM) keys that were in use by the Zendesk App Market (ZAM) service. These keys were disabled as part of a new key rotation automation process. This revocation caused the Zendesk App Market service to lose access to the public cloud storage resource bucket, resulting in errors when generating responses for the critical Apps API endpoint.

Resolution

The issue was resolved by reinstating the keys. This action restored access to the public cloud storage resource bucket and allowed the Zendesk App Market service to function correctly again.

Remediation Items

1. Replace IAM user access to the public cloud storage resource buckets for the ZAM service.
2. Disabled automated key revocation and work on implementing a manual process instead.
3. Create additional alerts to monitor diagnostic endpoints.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.