Resolved -
From 8:32-9:02 UTC, approximately 9.1M requests (77% of total) failed to the GCP us-east4 region of turbopuffer. Request failures were caused by load-balancers getting overwhelmed by traffic, we scaled up our load balancers to provide immediate relief. Currently, we are working on longer term preventative measures.
Mar 18, 11:21 UTC
Update -
The region has fully recovered. We continue to monitor for further issues.
Mar 18, 09:40 UTC
Monitoring -
We've deployed some mitigations to address the issue. API requests are recovering. We're continuing to monitor.
Mar 18, 09:26 UTC
Investigating -
We are currently investigating this issue.
Mar 18, 09:03 UTC