Loading…
Topic
110 posts about Kafka. Every summary links to the original.
See the idea
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Kafka is a durable event stream; Temporal is a workflow orchestrator. Learn which solves your problem and why many teams use them together.
Kafka is a distributed event streaming platform that ingests, buffers, and distributes high-volume data streams in real time, enabling applications to publish, subscribe, and process continuous event flows reliably.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.
Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.
Grab redesigned its messaging infrastructure from GrabChat to Message Center to overcome two-party chat limitations and support complex superapp requirements like group conversations and varied user roles. The architecture separates core processing logic from message delivery by splitting the system into a backend processor and an independently scalable postman service. Communication relies on an in-house TCP gateway named Hermes that proxies client payloads via gRPC, alongside Apache Kafka streams and Amazon SQS delay queues. Custom client-server acknowledgements and a DynamoDB event store ensure reliable message delivery even across dropped mobile TCP connections and offline reconnects.
Grab's streaming data platform team migrated the Kafka Control Plane from Role-Based Access Control to Attribute-Based Access Control to eliminate operational bottlenecks and manual permission management. The previous model required defining hundreds of roles, permissions, and group mappings in an internal IAM service, leading to approval delays and stale memberships. Under the new architecture, user attributes sync from the HRMS and token payloads, while resource attributes are tagged upon creation or backfilled to reflect department and team ownership. Open Policy Agent evaluates access requests defined in Rego via middleware by comparing user attributes with resource metadata. This transition eliminated over 200 roles, 200 permissions, and roughly 3,000 unused IAM resources while automating access provisioning for new joiners.
Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.
Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
Shopify’s Flex Comp lets employees allocate total rewards among base salary, Restricted Stock Units (RSUs), and Stock Options, with allocation changes during windows and no one-year equity cliffs. Building the system required a prioritized backlog, cross-functional teams, early API contracts, and privacy controls around its code and data. For sensitive analytics events, the team encrypted entire Kafka payloads with RSA, using 4096-bit keys for ordinary events and 6144-bit keys for larger payloads, while later identifying hybrid encryption as a better design. On the frontend, typed hooks, mocked data, and artificial delays allowed business logic and interface work to proceed in parallel while contracts evolved. Version one shipped to Shopify’s employees, with documented technical and design debt and future decisions retained for subsequent iterations.
Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.
Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.
Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.
Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates.
To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces.
Grab transitioned from an off-the-shelf MVP to an in-house ad serving system to accommodate business scale, hyperlocal requirements, and machine learning personalization. The architecture orchestrates core microservices and data pipelines across sequential steps: targeting, capping, pacing, scoring, ranking, pricing, and tracking. ElasticSearch serves as the targeting ads repository, while ScyllaDB acts as the high-throughput stats store fed by Kafka streams and data pipelines. The system operates on key engineering principles including parallelization and tuned latency limits, graceful fallbacks for slow dependency calls, and a unified server serving all ad types across the superapp.
Shopify built a real-time buyer-signal pipeline for Shopify Inbox after noting that 70 percent of its conversations involved customers making a purchasing decision. The system surfaces cart actions and order-completion information as chronological conversation events, covering events from up to 14 days before a conversation and post-conversation events while its seven-day lifecycle remains active. It uses Kafka through Monorail and Change Data Capture, Apache Beam on Google Cloud Dataflow, filtering jobs for mission-critical transactional events, and stateful aggregation with timers and a Global Window to retain buyer activity. In an equal-group A/B test, buyer signals increased merchant response rate by two percentage points and attributed conversion rate by 0.7 percentage points, while response time showed no significant change.
The best engineering reads, once a week.
