Previous incidents

We are up now.

RCA

• We use a third-party library to acquire distributed locks that expect specific LUA scripts to be cached. At 6:00 AM PT today, we realized that the Redis cache was burst due to disc corruption that led to the deletion of these scripts.
• We learned that the lib does not reindex the scripts, so we had to update them manually - once updated system is working as expected

API Service recovered.

The issue is mitigated. We are now working on a long-term fix.