Service Incident - July 9, 2024 - Guide | All Pods - Saved updates publishing issues

SUMMARY

On July 9, 2024, at 11:40 AM (UTC), we implemented a new change to our article editing system. This change inadvertently caused some published articles to be republished without user intervention, resulting in some draft versions of articles to be inadvertently published.

Timeline

July 09, 2024 08:41 PM UTC | July 09, 2024 01:41 PM PT
Our engineers have identified the issue and rolled back the affecting change across all pods. Guide article editing, saving, and publishing have been restored to normal behavior. Please note: If you saved any article edits between 11:40 UTC to 20:37 UTC, you may have inadvertently published draft article content. You will need to manually restore the previous published version by either using the Guide article revision history feature or manually updating the article and republishing.

July 09, 2024 07:57 PM UTC | July 09, 2024 12:57 PM PT
We have confirmed an issue with editing published Guide articles wherein clicking "save" inadvertently publishes the article live instead of only "saving" the current article draft version. We are working on reverting a change to restore the normal behavior. We will provide another update in 30 minutes or when have new information to share.

POST-MORTEM

Root Cause Analysis

An update intended to improve our editing system inadvertently caused edits to be processed twice, once through the old system and once through the new one. This led to articles being republished unnecessarily.

Resolution

Upon identifying the issue, we implemented a fix to disable the problematic feature. The system was stabilized, and normal operations resumed.

Remediation Items

We have identified several steps to prevent similar incidents in the future:

1. Ensure commands within our system are side-effect free.
2. Separate the republishing logic for different customer tiers.
3. Enhance our pre-deployment testing to better capture such issues.

We apologize for any inconvenience this may have caused and appreciate your understanding as we strive to improve our services.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.