We are closing this incident after 30 hours of stable monitoring with no further errors observed.
The incident caused degraded API performance and intermittent errors affecting some requests. We identified and addressed several contributing factors and applied infrastructure-level mitigations to stabilize the platform during the investigation.
Given the intermittent nature of the issue and its inability to be reproduced outside of production, we kept the incident open to ensure we had sufficient confidence in the stability of the platform before closing. Over the past 36 hours, all systems have been operating normally.
We will continue to monitor closely and are working on longer-term improvements to prevent recurrence. We apologize for the impact this caused and thank you for your patience.
Resolved
We are closing this incident after 30 hours of stable monitoring with no further errors observed.
The incident caused degraded API performance and intermittent errors affecting some requests. We identified and addressed several contributing factors and applied infrastructure-level mitigations to stabilize the platform during the investigation.
Given the intermittent nature of the issue and its inability to be reproduced outside of production, we kept the incident open to ensure we had sufficient confidence in the stability of the platform before closing. Over the past 36 hours, all systems have been operating normally.
We will continue to monitor closely and are working on longer-term improvements to prevent recurrence. We apologize for the impact this caused and thank you for your patience.
Identified
We are continuing to experience elevated API latency caused by high database pressure. Our team is actively working to mitigate the impact and will provide further updates as the situation evolves.
Identified
We have identified the root cause of the degraded performance affecting the API. The issue is caused by an unoptimized database query impacting response times. We have scaled our infrastructure to reduce the impact while a fix is being developed and deployed. We will provide a further update once it is live.
Investigating
We are experiencing elevated latency on our API endpoints. Some API requests may take longer than usual. Our team is investigating and working to resolve this. We will provide updates as we learn more.
