Loading…
What is Kafka?
Kafka is a distributed event streaming platform that ingests, buffers, and distributes high-volume data streams in real time, enabling applications to publish, subscribe, and process continuous event flows reliably.
Explanation
Kafka is a distributed event streaming platform designed to handle massive volumes of continuous data flows reliably and at scale. It sits between data producers and consumers, acting as a persistent buffer that allows applications to publish events independently of how quickly subscribers can process them, enabling each side to operate at its own pace. This decoupling is essential for modern data infrastructure because it prevents bottlenecks when downstream systems are slow or temporarily unavailable, allowing organizations to process millions of events per second without data loss. Real-time observability platforms, analytics pipelines, and event-driven applications depend on Kafka's ability to guarantee delivery and maintain data order, making it foundational for operations where timely and dependable event flow directly impacts business decisions.
Related reading
How Suprema Gaming made its data platform agent-ready with ClickHouse Cloud
Suprema Gaming migrated its analytics platform from Snowflake to ClickHouse Cloud on AWS as part of a company-wide move toward agentic operations. The previous stack delivered data about four hours late, required heavy dbt rebuilds, and often took minutes to answer queries, limiting real-time investigations and interactive analysis. Suprema rebuilt ingestion with ClickPipes for supported sources and Debezium plus Kafka for others, while dbt and GitHub Actions deploy the warehouse-layer structure without external cron jobs, Airflow, or a separate orchestrator; governed access uses role-based controls and query-time PII masking. Before cutover, every metric was validated against the reference source, and the resulting platform reduced warehouse spend by roughly 62%, improved direct-read performance 27x, and maintains about one-minute p95 freshness, supporting an early production deployment of ClickHouse Agents.
Ensuring reliable OpenTelemetry ingestion at scale
LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.
Shopify powers observability for global-scale commerce with ClickHouse
Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.
