Previous incidents

AI Models recovered.

All services are up. One node from the cluster went down bringing too many reservation loops.

We have mitigated this issue.

Quick RCA -

• CPU for one of our real-time DBs was getting throttled even if we had allocated more CPU. This is now fixed, and things should start working normally.
• We are keeping an eye on the overall setup

We are up now.

RCA

• We use a third-party library to acquire distributed locks that expect specific LUA scripts to be cached. At 6:00 AM PT today, we realized that the Redis cache was burst due to disc corruption that led to the deletion of these scripts.
• We learned that the lib does not reindex the scripts, so we had to update them manually - once updated system is working as expected

API Service recovered.

The issue is mitigated. We are now working on a long-term fix.