From 8:32-9:02 UTC, approximately 9.1M requests (77% of total) failed to the GCP us-east4 region of turbopuffer. Request failures were caused by load-balancers getting overwhelmed by traffic, we scaled up our load balancers to provide immediate relief. Currently, we are working on longer term preventative measures.
Posted Mar 18, 2025 - 11:21 UTC
Update
The region has fully recovered. We continue to monitor for further issues.
Posted Mar 18, 2025 - 09:40 UTC
Monitoring
We've deployed some mitigations to address the issue. API requests are recovering. We're continuing to monitor.