Ensuring reliable OpenTelemetry ingestion at scale

Summary

LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.

Context

The ingestion pipeline needed to absorb temporary mismatches between telemetry production and ClickHouse capacity, including arbitrary database backpressure and short-lived spikes, without losing data, delaying fresh telemetry behind a backlog, or permanently provisioning for peak demand. The team also wanted to avoid operating an additional external queuing system and keep costs controlled.

Approach / What changed

The architecture uses blob storage as durable overflow storage and the OpenTelemetry failover connector to keep ClickHouse as the primary destination while routing data to a secondary S3 destination when failures occur. Overflow notifications and a catchup collector support asynchronous recovery after ClickHouse returns. The design removed PVC-backed local WALs and was tested during a staging-region outage.

Takeaways

• A 1 TiB local collector WAL backlog took about four hours to drain, and FIFO consumption delayed fresh telemetry behind older data.
• During a staging outage processing 120,000 events per second, failover sent logs to S3 without changing gateway resource usage.
• The architecture scaled from 10 million to 50 million events per second, but adds regional bucket, queue, and event-notification setup and makes recovery less precise about which records remain outstanding.

Related reading