Loading…
Topic
110 posts about Kafka. Every summary links to the original.
See the idea
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Kafka is a durable event stream; Temporal is a workflow orchestrator. Learn which solves your problem and why many teams use them together.
Kafka is a distributed event streaming platform that ingests, buffers, and distributes high-volume data streams in real time, enabling applications to publish, subscribe, and process continuous event flows reliably.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
KIP-1279 introduces native cluster mirroring, embedding cross-cluster replication directly into the Kafka broker. This eliminates external tooling, offset translation tables, and recompression, as destination brokers fetch committed records using the standard fetch protocol and append them byte for byte. The approach preserves offsets, compression, and consumer group state, simplifying failover to stopping the mirror and redirecting clients. The architecture uses mirror fetcher threads inside the broker, extending AbstractFetcherThread, and leverages existing client quota mechanisms. Migration involves creating a new KRaft cluster, setting up a mirror, and starting mirroring, with future work including synchronous mirroring and tiered storage integration.
MakeMyTrip needed sub-second personalization for “last-searched” hotels across its consumer and corporate travel businesses, serving millions of daily users. Spark Structured Streaming’s micro-batch mode consistently delivered one to two seconds, while adopting Flink would have added a second engine, duplicated logic and increased operational overhead. The team instead adopted Databricks Real-Time Mode (RTM), using continuous data flow, concurrent pipeline scheduling and streaming shuffle to process events with millisecond-level latency on Spark. Its unified pipeline merges B2C and B2B Kafka streams, enriches events with each user’s “Last N” hotels from Aerospike, and writes results to Redis for serving in under 50ms. After tuning partitions, backpressure and rate limits, the production deployment reduced P50 latency from approximately 1.23 seconds to 44ms, P99 latency from over one minute to about 500ms, and increased click-through rate by 7%.
A Kafka topic configured with retention.ms=43200000 (12 hours) retained messages for four days because the active log segment never rolled. Kafka retains data by deleting whole closed segments, not individual records, and an active segment remains writable and cannot be deleted. Segment rolling occurs when segment.bytes or segment.ms is reached; in this case, traffic was too modest to fill the 1-GiB segment and only four days had passed, short of the seven-day default. Once a segment closes, retention.ms is evaluated against its newest record, followed by the retention check interval and file.delete.delay.ms before physical deletion. The proposed fix is to set segment.ms to a fraction of the retention window (one hour for the 12-hour example), while tuning segment.bytes as needed; a smaller rolling cadence makes effective retention more predictable.
Suprema Gaming migrated its analytics platform from Snowflake to ClickHouse Cloud on AWS as part of a company-wide move toward agentic operations. The previous stack delivered data about four hours late, required heavy dbt rebuilds, and often took minutes to answer queries, limiting real-time investigations and interactive analysis. Suprema rebuilt ingestion with ClickPipes for supported sources and Debezium plus Kafka for others, while dbt and GitHub Actions deploy the warehouse-layer structure without external cron jobs, Airflow, or a separate orchestrator; governed access uses role-based controls and query-time PII masking. Before cutover, every metric was validated against the reference source, and the resulting platform reduced warehouse spend by roughly 62%, improved direct-read performance 27x, and maintains about one-minute p95 freshness, supporting an early production deployment of ClickHouse Agents.
Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.
Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.
Bosch Mobility Platform Solutions developed the Logistics Operating System on AWS to unify vehicle tracking across fragmented spot logistics markets. The system standardizes multiple telematics providers, disparate data formats, and diverse communication protocols into a centralized visibility layer. Orchestration is handled by a Tracking Connector running on Amazon ECS with AWS Fargate, while AWS Lambda functions act as provider-specific adapters for protocol translation. Event-driven message ingestion and streaming rely on Amazon MSK, complemented by Amazon ElastiCache for low-latency caching and Amazon DynamoDB for compliance policy enforcement. Currently operational in India with ten integrated ISVs, the platform reportedly reduces integration costs by 15–20% and lowers total tracking costs for small transporters by 25–30%.
Uber’s Kafka deployment grew from 1 million to 12 million messages per second, but using Kafka’s ordered, partition-bound consumer model for asynchronous queueing created partition-scalability limits and head-of-line blocking. A one-second RPC example would require 1,000 partitions to reach 1,000 events per second, while non-uniform latency and poison-pill messages could stall unrelated work. Consumer Proxy addresses the mismatch by fetching messages through Kafka’s binary protocol, pushing them individually over gRPC, collecting processing statuses, and committing offsets only when safe. It also provides features including out-of-order commits, dead-letter-queue operations, flow control, and circuit breaking while hiding Kafka consumer-group management from services. At Uber, Consumer Proxy became the primary async-queueing solution, with hundreds of services onboarded; adaptive flow control and Kafka cluster federation are planned next steps.
Uber’s Payments Platform, internally called Gulfstream, grew from a small 2016 microservices effort into infrastructure supporting $217 billion in annualized gross bookings, nearly twice that amount in total money movement, and ledger balances for more than 1.2 billion entities. It is built around immutable Money Orders, zero-sum double-entry accounting, strongly consistent ledger balances, and three core models: money orders, ledgers, and entity changelogs. Independent stateless services communicate primarily through Apache Kafka, while generic abstractions for lines of business and payment instruments let new products and methods plug into the platform; synchronous in-session flows use Cadence. As scale and audit requirements increased, Uber introduced Entity Fares, addressed hot ledger entities, and built LedgerStore for auditability and tamper-evident record-keeping, while also extending the ledger as a multi-tenant service.
Etsy uses Kafka streams to provide fresh features and embeddings for machine learning models, which infer shopper intent and listing meaning from visits and listing data. Previously, engineers manually assembled Java and YAML applications, selected schemas and base classes, and handled testing and local execution. Etsy introduced conversational Kafka Skills that use language models to interpret requests, pass structured inputs into versioned scaffolding and validation tools, run applications locally, and open pull requests. Telemetry records Skill invocation and pull-request completion through protobuf messages in Kafka and BigQuery, while least-privilege ACLs, repository permissions, and deployment controls constrain agent actions. Initial adoption was low because developers did not know the Skills existed, so Etsy added vocabulary-specific and repository-level entry points; it now has Skills for seven Kafka workflows and is focusing on discoverability.
NVIDIA DeepStream 9.1 introduces AutoMagicCalib and Multi-View 3D Tracking skills to resolve the limitations of single-camera 2D tracking and manual camera calibration in large-scale video analytics. Multi-View 3D Tracking projects detections into a shared 3D coordinate system, matching tracklets across cameras via MQTT and maintaining globally consistent object IDs. AutoMagicCalib automates camera intrinsic and extrinsic parameter estimation by analyzing object trajectories across video feeds, optionally using a Visual Geometry Grounded Transformer. These capabilities are integrated into 13 agentic skills that allow coding agents to set up, configure, and execute end-to-end vision AI pipelines from natural language prompts. Outputs include an on-screen display grid, bird's-eye view trajectories, and per-frame protobuf metadata streamed through Kafka.
Bullet operates a DeFi exchange for perpetuals, spot trading, and lending, with trades settled on a blockchain while users retain custody of their assets. Its sub-millisecond trading system produces more than 150 million event rows per hour, creating a need for historical indexing, real-time analytics, and operational monitoring beyond the blockchain’s state-machine model. ClickHouse Cloud consumes events from Kafka, stores them in dedicated tables, and supports user features, leaderboard and reward calculations, P&L charts, audit logs, and Grafana-backed alerts. Replacing Databricks reduced query latency from 10–15 seconds to milliseconds, improved data freshness from 1–2 hours to under 5 seconds, and removed a separate DynamoDB serving layer at comparable cost while handling 1,000 times more data across staging, testnet, and mainnet.
Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.
The post presents a reusable real-time guardrail pattern for flagging suspicious Ethereum blockchain transactions and routing them for downstream action. Its rules identify impossible blocks where gas_used exceeds gas_limit and scan extra_data for email addresses, JWT tokens, or AWS access-key patterns, producing ALLOW or QUARANTINE decisions with reasons. The implementation uses Apache Spark Structured Streaming Real-Time Mode, whose continuous data flow, pipeline scheduling, streaming shuffle, pre-allocated execution pipelines, and asynchronous checkpointing target millisecond latency without a separate streaming engine. On a four-worker DBR 16.4 LTS cluster, the stateless Kafka-to-Kafka test processed about 23 million records at 69,713 rows per second, with P95 below 0.5 milliseconds and P99 at 1 millisecond. The post notes that RTM with a Kafka sink provides at-least-once delivery and that more complex stateful workloads may incur higher latency.
Feature Views are presented as a framework for creating managed feature pipelines in Databricks, addressing duplicated logic, training/serving skew, backfills, online-store plumbing, and governance for real-time ML. A Feature View defines the source, entity, time-series column, and computation once; Databricks uses it to generate point-in-time-accurate historical data, then materializes pipelines for batch or streaming production use. Registered in Unity Catalog, materialized features receive lineage and access controls, while MLflow records model dependencies and Model Serving retrieves required features without custom lookup code. For Kafka-sourced streaming features, Spark Realtime Mode, Lakebase, and Model Serving support rolling-window updates with reported end-to-end p99 latency of 200ms from event to online availability. The announcement says Feature Views are in Public Preview and can be developed through the Feature Engineering Client SDK and Genie Code, with streaming materialization requiring an Enterprise-tier workspace in a region supporting Lakebase.
Uber describes how it uses early chargeback signals, especially TC40 reports, to identify payment-fraud patterns before conventional chargebacks arrive. These notifications come from payment service providers through fraud reports and are ingested from external vendors via SFTP, third-party APIs, or webhooks, then validated, normalized, published to Apache Kafka, and stored in Apache Hive. A streaming pipeline maps signals to internal transactions and users, derives near-real-time features, and sends them to machine-learning models and a rule engine that determines risk actions. Redis TTL caching filters duplicate references, while Kafka propagates signals to downstream systems and Hive retains them for batch processing. Since launch, the project has processed more than 2 million additional signals, and TC40 signals arrive on average 4–5 days earlier than chargeback signals, although the source notes that TC40 data alone does not prevent chargebacks.
Verihubs rebuilt its data warehouse after a Postgres-based, daily-batch pipeline became slow, unreliable, and difficult to correct when late vendor callbacks required backfills. The new architecture uses Kafka, Kafka Connect, and Debezium for real-time ingestion into ClickHouse, with MergeTree for append-only logs and ReplacingMergeTree for datasets updated through change data capture. Materialized views create daily and monthly rollups, projections accelerate existing query patterns, and Dagster handles complex detection workflows that can notify Slack. The migration, completed in about six months, reduced query times from 20–30 minutes to two or three seconds on datasets up to 18 million rows, representing as much as 98% faster queries and 50% lower infrastructure costs. Verihubs says the resulting real-time OLAP system gives finance and operations fresher data for traffic analysis, reconciliation, performance reporting, and invoicing.
Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces _timestamp or _partition_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control.
Uber introduces uForwarder, an open-source Consumer Proxy for Kafka Async Queue workloads, after deploying a push-based system across more than 1,000 consumer services. The proxy fetches messages through Kafka’s binary protocol, sends them individually to consumer services over gRPC, collects status codes and processing results, and commits offsets when safe. Production challenges included head-of-line blocking from messages that fail before reaching handlers, growing hardware costs, external data sources that are not ready when events arrive, and requirements for production/non-production and zone isolation. The implemented solutions include context-aware routing through message and gRPC headers, active blocking detection and mitigation, hardware-efficiency improvements, and DelayProcessManager, which pauses only delayed partitions and guarantees at-least delay rather than exact timing. The project is open source, with planned work on consumer-lag handling and direct Protobuf support.
Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.
The best engineering reads, once a week.
