Summary
On March 7, 2025 from 10:44 UTC to 23:19 UTC, some Zendesk Guide customers on multiple pods experienced corrupted Guide content for Korean and Japanese articles that appeared as garbled characters in search results.
Timeline
March 10, 2025 04:28 AM UTC | March 09, 2025 09:28 PM PT
We have received reports of garbled characters in Guide search results and articles for Japanese content on Pod 25. We are currently investigating the issue. We will provide an update when we have more information to share.
March 10, 2025 04:52 AM UTC | March 09, 2025 09:52 PM PT
We have identified and fixed the issue impacting Korean and Japanese articles and search results in Guide. Please refresh the impacted articles/searches and try again. We will provide a final update when we confirm full resolution of the issue.
March 10, 2025 05:53 AM UTC | March 09, 2025 10:53 PM PT
A full backfill is currently running to fix all remaining issues on Pod 25. If your account is on a different Pod, please rest assured that our team is working to restore the correct search and article results on all Pods. We will post a final update when all backfills are finished.
March 10, 2025 11:32 AM UTC | March 10, 2025 04:32 AM PT
We would like to provide you with an update regarding the ongoing efforts of our engineering team to address the issue of garbled characters in Guide search results and articles related to Japanese content. We are pleased to inform you that a fix has been implemented, which ensures that newly created or edited content will no longer experience this issue. Additionally, the team is actively working on backfilling this fix for existing content. While progress on Pod 25 is still underway, the team has successfully completed the necessary work on content hosted on Pod 26 and has now commenced efforts on Pod 27. They will continue to address this issue across all affected Pods. Thank you for your understanding and support as we work to resolve this matter.
March 10, 2025 03:04 PM UTC | March 10, 2025 08:04 AM PT
Our engineers continue their work on backfilling the fix for all of the impacted Pods for the issue with garbled Korean and Japanese characters rendering garbled search results and articles. So far the issue has been amended on Pods 13, 15, 17, 19, 20, 26, 27 and 29. With Pods 18, 23 and 25 still being in progress.
March 10, 2025 05:56 PM UTC | March 10, 2025 10:56 AM PT
The backfill has completed, however we are still catching up in some of the affected pods. New articles and article updates may still experience some delays. We will update when we have fully recovered.
March 10, 2025 11:26 PM UTC | March 10, 2025 04:26 PM PT
We are happy to report that the backfills have been completed for the affected Korean and Japanese Guide articles. Thank you for your patience during this extended incident.
Root Cause Analysis
This incident was caused by a backend service update which changed the parsing used in our Guide indexing process. The documents indexed between the release and rollback of the update were impacted by this change.
Resolution
To fix this issue, our team identified the code that was updated and rolled it back to the last working version. Further work was done to backfill and correct older articles that were impacted during the incident.
Remediation Items
- Implement smoke and unit tests to detect potential article corruption in future updates.
- Investigate improvements and alternative solutions for future service updates.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.
0 comments
Article is closed for comments.