Postmortem:

A failover of one of the Kafka brokers triggered a cascading effect on an event queue, which led to degraded dashboard rendering and increased latency on some API endpoints.

We fully recovered within 3 minutes, but the system experienced intermittent degradation for the following 20 minutes. To prevent recurrence, we have added an extra replica and increased pod affinity.

We apologize for the inconvenience.

Dashboard recovered.

Dashboard went down.