Previous incidents

Update

The RCA is tracked on the following Google public issue: https://issuetracker.google.com/issues/363324206

They have enabled a liveness probe on the cluster to ensure this doesn't happen again. We are taking some precautionary steps to avoid this entirely in the future.

AI Models recovered.

All services are up. One node from the cluster went down bringing too many reservation loops.

We have mitigated this issue.

Quick RCA -

• CPU for one of our real-time DBs was getting throttled even if we had allocated more CPU. This is now fixed, and things should start working normally.
• We are keeping an eye on the overall setup

We are up now.

RCA

• We use a third-party library to acquire distributed locks that expect specific LUA scripts to be cached. At 6:00 AM PT today, we realized that the Redis cache was burst due to disc corruption that led to the deletion of these scripts.
• We learned that the lib does not reindex the scripts, so we had to update them manually - once updated system is working as expected

API Service recovered.