Resolved -
Between 9:20-9:23 UTC turbopuffer returned HTTP 500 errors to upsert and query requests in our gcp-us-central1 cluster due to a bug in newly deployed load balancing code, which was reverted immediately.
Dec 11, 09:00 UTC
Resolved -
Between 20:00 and 01:04 UTC 5.5% of namespaces in gcp-us-central1 may have experienced latency to the point that it may have caused timeouts. The root cause was a maxed out NVMe volume on a single query node in gcp-us-central1. The team's taking additional actions to address the root cause of the high NVMe utilization.
Dec 1, 20:00 UTC