Resolved -
Between 14:20 and 14:35 (UTC), a cluster maintenance operation caused about 19% of requests in our gcp-us-central1 cluster to fail with 429, 500, or 502 errors. The issue has been resolved.
Nov 20, 14:20 UTC
Resolved -
Between 17:18 UTC and 17:24 UTC the region `gcp-us-central1` served errors for 99-100% of request. The problem has been resolved.
A change made to our deployment configuration files triggered an edge case in our deployment tooling, which took the API server deployment offline entirely. Our monitoring tools alerted us immediately and the change was reverted. We have added a safeguard to prevent any issues in the deployment tooling from triggering destructive actions.
Nov 19, 17:27 UTC