Service Incident - May 13, 2025 - Multiple Pods | Agent AI - Customers unable to log in to Ultimates AI Agents, receiving “Couldn’t authenticate user” errors

Summary

On May 13, 2025, from 09:53 UTC to 13:29 UTC, customers in the EU region experienced login failures when accessing their Ultimate AI Agents accounts via Zendesk’s admin center. Users encountered “Couldn’t authenticate user” errors that prevented access to the AI Agents dashboard. Additionally, some billing events related to AI Agents were not processed correctly due to the same underlying issue. While the AI Agents themselves remained functional for end users, the login and billing disruptions impacted customer experience and reporting.

Timeline

May 13, 2025 12:58 PM UTC | May 13, 2025 05:58 AM PT

We have received reports of customers unable to login to their Ultimates AI accounts and getting a message saying 'Couldn't authenticate user'. Our engineers are investigating the source of the issue. We apologize for the inconvenience and ask for your patience as we work to resolve this issue. We will provide updates as it becomes available.

May 13, 2025 01:29 PM UTC | May 13, 2025 06:29 AM PT

We are pleased to report that the issue where customers are unable to login to their Ultimates AI accounts has been mitigated. Our engineers have applied the fix and are monitoring the system activity. We would like to ask for your cooperation to test if you are now able to get back into your Ultimate AI accounts and reply back to us with the results. Your quick responses will be highly appreciated.

Root Cause Analysis

The incident was caused by clock drift and network latency issues affecting certain system components responsible for user login and billing processes. These issues led to failures in verifying user access and disruptions in billing event reporting. Additionally, a cache refresh issue was identified that impacted billing accuracy when specific settings were changed.

Resolution

To fix this issue, we redeployed the affected services onto stable nodes with synchronized clocks and improved network reliability. Additionally, we implemented a fix to resolve the cache refresh problem. These measures restored normal login functionality and ensured accurate billing event processing.

Remediation Items

1. Deploy the permissions service only on reliable nodes with consistent timing to prevent future access verification problems.
2. Implement monitoring and alerting for authentication and billing event errors to enable faster detection and response.
3. Fix cache management to ensure billing events are accurately tracked when organizational changes occur.
4. Continue enhancing error tracking and reporting for AI Agents services.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.