Previous incidents

This is now resolved, and ingestion is back to its original speed.

The node is recovered. Ingestion is resumed.

We’ve received an update from the Clickhouse team. Here’s the crux of the issue:

"Due to memory starvation, other processes in your cluster are starting to fail, resulting in degraded performance, likely including the failed writes you are experiencing."

All queued logs have been written to the disk and the instance is back to normal.

We’re still keeping an eye on our data pipelines. We’ve also put together a checklist with the team to help prevent this from happening again.

The latency is back to normal. We will be keeping an eye on the system for next few hours.

Postmortem:

A failover of one of the Kafka brokers triggered a cascading effect on an event queue, which led to degraded dashboard rendering and increased latency on some API endpoints.

We fully recovered within 3 minutes, but the system experienced intermittent degradation for the following 20 minutes. To prevent recurrence, we have added an extra replica and increased pod affinity.

We apologize for the inconvenience.

API Service recovered.