Loading…
Topic
110 posts about Kafka. Every summary links to the original.
See the idea
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Kafka is a durable event stream; Temporal is a workflow orchestrator. Learn which solves your problem and why many teams use them together.
Kafka is a distributed event streaming platform that ingests, buffers, and distributes high-volume data streams in real time, enabling applications to publish, subscribe, and process continuous event flows reliably.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Spyne’s CDC pipeline streamed changes from MySQL and MongoDB through self-managed Debezium, Kafka, and ClickHouse, but table onboarding required 1 to 1.5 hours of manual work and schema drift could silently exclude new columns. The team migrated to ClickPipes and ClickHouse Cloud, where pipelines can be configured in minutes and automated schema evolution applies upstream changes to corresponding ClickHouse tables without manual intervention or restarts. To manage snapshot pauses, Spyne isolated about 10 critical historical tables in dedicated ClickPipes and grouped lower-priority tables separately. The migration also required using FINAL in downstream queries to handle ReplacingMergeTree eventual consistency, while binlog gaps may still require full table resynchronization and higher costs. Spyne reports less operational toil, faster feature and analysis workflows, and improved data consistency, accepting reduced recovery control and query-time CPU trade-offs.
METRO Markets outgrew its self-hosted Impala and Kudu-based Hadoop stack as the marketplace expanded, with parallel stores, inconsistent reporting, and too much maintenance burdening a lean data team. They evaluated BigQuery, Snowflake, and ClickHouse against production queries, finding faster ClickHouse performance, including one response falling from four seconds to under 200 milliseconds, while its pricing supported more predictable costs. Airbyte moves CDC data from MySQL and Postgres into ClickHouse; ReplacingMergeTree handles deduplication during background merges, and BigQuery-hosted Google Analytics data enters through cloud storage and external tables. The resulting warehouse consolidates company data and supports seller analytics, credit risk features through Query API endpoints, observability, and emerging AI use cases, while teams query a shared dataset through compatible tools; planned Kafka and Flink integration would extend it to streaming workloads.
Appcues replaced an analytics stack that included Snowflake and Airflow after rising scale pushed queries to 20 seconds or more and limited reports to 10-minute windows. The company evaluated ClickHouse Cloud on AWS for real-time segmentation, exporting Snowflake data to Parquet in S3 and testing complex email segments with nested conditions, historical events, and negative logic. Production events now pass through Kafka and a clickwriter consumer, while a clickclient library built on Elixir’s Ecto generates controlled ClickHouse queries; SharedReplacingMergeTree, table engines, and materialized views were tuned during a one-year migration. Across 1.31 PB and 410 billion events, P95 queries fell below two seconds, ingestion latency dropped to about five seconds, and analytics costs decreased 23%, while new funnel reports and AI-powered insights were added.
Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.
Hugo evolved from a batch-oriented Spark platform into Grab’s unified, self-service data ingestion system for near real-time pipelines. The earlier expansion across Kafka Connect, Sprinkler, and Hugo fragmented onboarding, forcing users to translate configurations and manage manual schema changes across systems. A custom automation layer and Apache Flink now orchestrate one-click MySQL CDC and Kafka ingestion, retiring Kafka Connect and Sprinkler for the data lake. For CDC, Flink reads the binlog directly, uses automated schema detection and dynamic validation, and writes to queryable Hive tables through Spark compaction; for Kafka, it fetches Protobuf schemas from Confluent Schema Registry at runtime and resumes from the latest checkpoint after restart. Onboarding now takes approximately three minutes for MySQL CDC and six for Kafka, although schema updates still require manual restarts; new pipeline adoption in the last year also exceeded the previous five years combined.
The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.
Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.
Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.
Distributed tracing frequently suffers from visibility gaps, broken context across asynchronous boundaries like Kafka, and high operational overhead from manually configuring OpenTelemetry SDKs and collectors. To resolve these issues, Odigos provides zero-code instrumentation via eBPF probes that dynamically attach to running workloads without modifying application code. An intelligent control plane within Odigos dynamically adjusts trace sampling and attribute enrichment based on real-time latency and error rates. The resulting telemetry exports over OTLP into ClickStack, an all-in-one backend featuring an OpenTelemetry Collector, ClickHouse storage, and the HyperDX query interface. This unified pipeline preserves end-to-end trace continuity, captures deep application and database context, and enables fast SQL-based trace analysis.
AI token spend is volatile, consumption-based, and distributed across teams, making usage and cost difficult to monitor. Ramp's pipeline ingests LiteLLM and OpenRouter events through authenticated, tenant-scoped webhooks, Kafka, and ClickHouse, then aggregates them for REST dashboards and AI-generated forecasts. LiteLLM uses callbacks with token counts and metadata, while OpenRouter sends OTLP traces; ReplacingMergeTree deduplicates replayed events using business_id, source, and event_id. Customers can connect gateways with an API key, configure LiteLLM callbacks or OpenRouter Broadcast, and pass attribution metadata. The resulting visibility supports breakdowns by provider, model, department, user, project, and use case, and exposed phantom Gemini reasoning tokens after a LiteLLM upgrade, geo pricing premiums, runaway loops, and prompt bloat.
Documentation platform Mintlify experienced major dashboard performance degradation and rate limiting under PostHog as multi-tenant analytics traffic grew rapidly. Because PostHog lacked incremental materialized views, queries scanned full raw datasets and took tens of seconds to load. Mintlify migrated its customer-facing analytics infrastructure to ClickHouse Cloud, ingesting events through Amazon MSK and ClickPipes into a single events table backing incremental materialized views. The team executed the transition over two weeks by dual-writing events and replicating the existing PostHog schema to preserve API surface compatibility. The migration reduced dashboard query latency to sub-one-second, decreased related support bug reports to zero, lifted NPS by an estimated 30%, and lowered analytics infrastructure costs by approximately 60%.
Notion’s data residency work keeps customer data processed and stored in the region where it was generated, supporting EU requirements across Search, analytics, scheduled events, and AI. To make this possible, Notion modularized infrastructure around isolated private networks and workspace IDs, then deployed regional data lakes, Kafka clusters, Spark jobs, Elasticsearch, and vector databases. Debezium captures Postgres changes into regional Kafka, while Airflow remains a centralized orchestration layer that routes jobs to regional EMR clusters; event logging similarly routes traffic by workspace and sanitizes data before Snowflake. Centralized mapping tables, shared region helpers, Terraform, and localized sanitization let teams retain a unified Snowflake environment without exporting customer data. The work took months, and Notion says it is applying these patterns to Japan and South Korea while aiming to provision future regional data stacks within days rather than months.
Notion describes two years of scaling vector search for AI Q&A, where semantic embeddings retrieve workspace and connected-tool content by meaning rather than exact keywords. The initial system combined Apache Spark batch indexing with Kafka-driven real-time updates, workspace-based sharding, and generation routing; rapid demand drove a 600x increase in daily onboarding capacity and cleared a waitlist of millions of workspaces by April 2024. A serverless migration cut vector database costs 50 percent from peak usage, while a later move to turbopuffer reduced search-engine spend 60 percent, lowered AWS EMR costs 35 percent, and improved p50 query latency from 70–100ms to 50–70ms. Further work uses xxHash-based page-state caching in DynamoDB to reprocess only changed spans and shifts embeddings generation from Spark and query embeddings away from a third-party API to Ray on Anyscale, with early rollout results pointing to a projected 90+ percent reduction in embeddings infrastructure costs.
Grab’s Customer Data Platform (CDP) introduced Scenarios to address the limits of daily batch attributes for real-time personalization across marketing and user journeys. The capability combines Scribe event triggers, geo-fencing, optional historical CDP data, and predictive models, with a self-serve GUI for configuring and validating event sequences and APIs for programmatic management. Incoming events are processed through Flink, with historical data evaluated in StarRocks and outputs written to Kafka, Amphawa, or both for downstream consumption; configured Scenarios become available within an hour and are designed for latency under 15 seconds. More than 12 implementations are live, including a Grab Unlimited abandonment intervention that notified users 15 minutes after signup drop-off and produced over a 3% uplift in subscriber conversions versus non-real-time acquisition campaigns. Planned work includes scalable Scenario sharding, multiple output destinations, and delayed triggering.
Kafka stream data can fail syntactically when actual records do not match expected schemas, or semantically when fields violate agreed patterns, ranges, or constants. The described platform addresses this gap with user-defined data contracts covering schema agreements, field-level semantic rules, stream configuration, and ownership metadata. An LLM recommends semantic rules from schemas and anonymized samples, while a transformation engine converts them into Test Runner configurations and inverse SQL queries. A long-running FlinkSQL Test Runner reads Kafka with its own consumer group, publishes issue events and stores them in AWS S3; Genchi sends Slack alerts and Coban shows violating fields and sample records. Deployed across 100+ critical topics, the solution provides real-time detection, alerting, and visibility, with future work targeting producer tracing, cross-field tests, completeness, freshness, and DTO change detection.
Shopify’s BFCM readiness program addresses the challenge of operating critical commerce infrastructure at traffic levels informed by 2024 records, while preparing APIs with no prior holiday-traffic history. Starting in March, teams combined capacity planning on Google Cloud, an infrastructure roadmap, risk assessments, chaos-engineering Game Days, and a Resiliency Matrix covering failure modes, recovery objectives, runbooks, and on-call coverage. Genghis load tests and Toxiproxy network faults ran across three GCP regions, while five April–October scale tests exercised browsing, checkout, analytics, webhooks, regional failover, and cascading failures; testing reached 146 million requests per minute and a p99 target of 200 million. The work exposed and addressed Kafka partition, memory, timeout, routing, queueing, and failover issues, while the operational plan adds monitoring, 24/7 incident coverage, merchant communications, live tuning, and a post-BFCM review.
Endianness dictates the order in which multi-byte numbers are stored and transmitted across systems. While big-endian formats store the most significant byte first at the lowest address, little-endian systems place the least significant byte first. Communication across diverse architectures relies on consistent byte ordering to prevent critical interpretation errors in fields such as telecommunications, IoT, automotive, finance, and aerospace. To safely process disparate payloads, modern ingestion pipelines deploy an endian converter to normalize raw binary data into a unified format. These normalized data streams flow through Apache Kafka, undergo real-time processing and enrichment via Atlas Stream Processing, and persist in MongoDB Atlas for downstream analytics and visualization.
Partner Gateway needed low-latency analytics for API traffic and response-time metrics across intervals from one hour to 30-day retrospectives, with a particular dataset potentially reaching approximately 6.8 billion rows in a 30-day period. Apache Pinot was selected as a distributed real-time OLAP datastore, using high-throughput ingestion and columnar, indexed, partitioned data for millisecond analytical queries. Initial aggregation queries on datasets over 150GB timed out for 20- to 30-day periods, exceeding 10 seconds and threatening a 300-millisecond SLA. The implementation routes metrics through Kafka and Apache Flink, then ingests Schema Registry-compatible messages into Pinot, while partitioning by metric name and adding aggregation-focused optimizations. For one metric, query latency fell from 5,072 to 35 milliseconds; Star-tree indexing improved speed but raised 30-day storage from 192GB to 373GB, with the additional AWS EBS cost described as minor.
MongoDB announced its 2025 Global Partner Award recipients to recognize organizations driving AI adoption, legacy system modernization, and collaborative market expansion. Microsoft earned Global Cloud Partner for Azure integrations, while Amazon Web Services took Global AI Cloud Partner after assisting Novo Nordisk in cutting a key workflow from twelve weeks to ten minutes with Amazon Bedrock. Google Cloud received the Global Cloud GTM Partner award, and Confluent was named Global Technology Partner with over 550 joint deployments. LangChain and Pureinsights received honors for their integrations supporting retrieval-augmented generation and search solutions. Additional awards celebrated Accenture, BigID, gravity9, IBM, and Alibaba Cloud for their enterprise and public-sector impact.
MongoDB announced the recipients of its 2025 Global Partner Awards to recognize partner contributions across cloud modernization, artificial intelligence, and enterprise integration. Microsoft received the Global Cloud Partner award for Azure integrations, while Amazon Web Services secured the Global AI Cloud Partner award following joint generative AI implementations such as reducing workflow durations for Novo Nordisk. Google Cloud earned the Global Cloud GTM Partner distinction through shared sales development programs, and Accenture took Global SI Partner honors after establishing a dedicated engineering Center of Excellence. Confluent was named Global Tech Partner with more than 550 joint streaming deployments, and LangChain was recognized as Global AI Tech Partner for vector search and agentic frameworks. The awards demonstrate partner-led technical solutions spanning distributed databases, data streaming pipelines, and scalable enterprise cloud architectures.
The best engineering reads, once a week.
