---
title: "Ensuring reliable OpenTelemetry ingestion at scale"
description: "LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility."
---

# Ensuring reliable OpenTelemetry ingestion at scale

[Clickhouse](https://yomu.fyi/company/clickhouse) · Aug 24, 2026

**Type:** Problem & solution

## Summary

LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.

## Context

The ingestion pipeline needed to absorb temporary mismatches between telemetry production and ClickHouse capacity, including arbitrary database backpressure and short-lived spikes, without losing data, delaying fresh telemetry behind a backlog, or permanently provisioning for peak demand. The team also wanted to avoid operating an additional external queuing system and keep costs controlled.

## Approach / What changed

The architecture uses blob storage as durable overflow storage and the OpenTelemetry failover connector to keep ClickHouse as the primary destination while routing data to a secondary S3 destination when failures occur. Overflow notifications and a catchup collector support asynchronous recovery after ClickHouse returns. The design removed PVC-backed local WALs and was tested during a staging-region outage.

## Takeaways

- A 1 TiB local collector WAL backlog took about four hours to drain, and FIFO consumption delayed fresh telemetry behind older data.
- During a staging outage processing 120,000 events per second, failover sent logs to S3 without changing gateway resource usage.
- The architecture scaled from 10 million to 50 million events per second, but adds regional bucket, queue, and event-notification setup and makes recovery less precise about which records remain outstanding.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Clickhouse](https://clickhouse.com/blog/reliable-opentelemetry-ingestion-at-scale)
- Source URL: https://clickhouse.com/blog/reliable-opentelemetry-ingestion-at-scale
- Ingested by Yomu: 2026-08-29T07:46:58.190Z

[Read original post](https://clickhouse.com/blog/reliable-opentelemetry-ingestion-at-scale)