All services are online

Last updated on Oct 10 at 04:09am IST

Website
Dashboard
API Service
AI Models
30 days ago
Today
SDK Playground
30 days ago
Today
Blog
30 days ago
Today

Previous incidents

Oct 04, 2024

Degraded availability (test runs, prompt playground)

Degraded

Resolved Oct 04 at 08:39pm IST

We are up now.

RCA

  • We use a third-party library to acquire distributed locks that expect specific LUA scripts to be cached. At 6:00 AM PT today, we realized that the Redis cache was burst due to disc corruption that led to the deletion of these scripts.
  • We learned that the lib does not reindex the scripts, so we had to update them manually - once updated system is working as expected

2 previous updates

Dashboard and API Service are down

Downtime

Resolved Oct 04 at 02:06pm IST

API Service recovered.

5 previous updates

Redis cluster upgrade

Maintenance

Resolved Oct 04 at 06:00pm IST

We are performing an urgent Redis upgrade that may cause approximately 5 minutes of degraded performance.