---
title: "Kafka"
description: "110 posts about Kafka, summarised, each linking to the original."
---

# Kafka
> 110 posts about Kafka, summarised, each linking to the original.

## Articles

### [Scaling Shopify's BFCM Live Map: An Apache Flink Redesign](https://yomu.fyi/post/scaling-shopify-s-bfcm-live-map-an-apache-flink-redesign.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 10, 2021
- Type: Problem & solution

Shopify’s 2021 BFCM live map had to scale from more than 1.7 million merchants while adding product trends and unique shoppers, after the prior Cricket-and-Redis backend showed volume and delivery weaknesses. The redesign placed Apache Flink on the critical path to filter irrelevant checkout events, reducing Cricket’s workload to about one percent of event volume, while cross-region sharding and active-active deployment supported availability. Flink computed new metrics, including 500 product categories ranked by sales-volume change every five minutes, while Cricket retained existing metric computation and deduplication. In load tests, the pipeline handled 1 million checkout events per second, and its jobs ran at 100 percent uptime during BFCM without backpressure or manual intervention. Redis still caused high CPU and memory pressure, producing brief arc-visual errors for a small sample of users, so Shopify plans a simpler Flink-centered design.


### [Search indexing optimisation](https://yomu.fyi/post/search-indexing-optimisation.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weilun Wu
- Published: Sep 27, 2021
- Type: Problem & solution

Grab Deliveries uses MySQL as canonical storage and Elasticsearch as derived search storage, with Food-Puxian synchronising updates through Kafka. The original incremental-sync design treated Kafka events mainly as notifications, reloaded records from MySQL, and used separate consumers and in-memory event buffering, creating heavy database traffic, data-loss gaps, coupling to table schemas, redundant Elasticsearch updates, and duplicate cascade updates. A CDC system based on MySQL row-based Binlog provides common events containing Operation, PayloadBefore, and PayloadAfter, enabling filtering of irrelevant changes, partial Elasticsearch updates, event merging in the buffer, and deduplication of cascade updates through a dedicated Kafka stream. The reported results were an 80% reduction in database load from the first optimisation and an 80% reduction in cascade-update events. Overall, DSP saved about 91% of database reads and 90% of Elasticsearch reads, while stream-consumer throughput rose from 200 to 800 QPS, with peaks above 1000 and lower synchronisation latency.


### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weicheng Zhu
- Published: Sep 1, 2021
- Type: Problem & solution

Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.


### [Shopify-Made Patterns in Our Rails Apps](https://yomu.fyi/post/shopify-made-patterns-in-our-rails-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 13, 2021
- Type: Explainer

Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.


### [High Availability by Offloading Work Into the Background](https://yomu.fyi/post/high-availability-by-offloading-work-into-the-background.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 8, 2021
- Type: Explainer

Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.


### [Deleting the Undeletable: Managing PII at Shopify - Shopify](https://yomu.fyi/post/deleting-the-undeletable-managing-pii-at-shopify-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2021
- Type: Problem & solution

Shopify describes how its Privacy and Data Science & Engineering teams redesigned analytical event handling to make personal-information deletion workable at Shopify’s scale. The effort addressed a lack of guaranteed event schemas by introducing a human-readable JSON schematization system covering field types, version evolution, ownership, data controllers and subjects, and privacy instructions; more than 4,500 active schemas now govern events. Before warehouse storage, identifying data is either obfuscated into generalized values, such as masked IP addresses with city or country information, or tokenized into consistent random values whose mappings reside in a secured vault and vary by controller and subject. Removing a vault mapping makes warehouse copies of the token non-detokenizable, allowing deletion without warehouse operations; the production platform processes about 20 billion events daily, while tens to hundreds of millions of mappings are deleted daily.


### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Warren Zhou
- Published: May 24, 2021
- Type: Problem & solution

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.


### [Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms](https://yomu.fyi/post/building-smarter-search-products-3-steps-for-evaluating-search-algorit.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 1, 2021
- Type: Tutorial

Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.


### [Capturing Every Change From Shopify’s Sharded Monolith](https://yomu.fyi/post/capturing-every-change-from-shopify-s-sharded-monolith.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 12, 2021
- Type: Problem & solution

Shopify’s separate internal warehouse and Merchant Analytics Platform created duplicated work, differing extraction results, and incompatible tooling while demand grew for fresher, low-latency merchant data. Longboat, its query-based CDC service, polls sharded-monolith replicas in batch jobs using updated\_at, but queries can run only hourly and cannot detect hard deletes or updates that bypass that field. Shopify therefore adopted Kafka and Debezium CDC to capture database changes, standardize extraction, and make the event stream usable by both streaming and batch consumers. CDC-backed Beam modeling reduced average freshness for one migration from one day to one hour, while Kafka became a shared source for operational and analytical data. Shopify still identifies incremental snapshotting and streaming-state problems, including reconciling MySQL table chunks with ongoing binlog events without expensive locks, as ongoing work.


### [Trident - Real-time Event Processing at Scale](https://yomu.fyi/post/trident-real-time-event-processing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Jan 13, 2021
- Type: Problem & solution

Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.


### [Apache Beam for Search: Getting Started by Hacking Time](https://yomu.fyi/post/apache-beam-for-search-getting-started-by-hacking-time.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 8, 2021
- Type: Tutorial

Search relevance can use clickstream signals such as clicks and purchases to promote popular results and demote results users do not favor, but batch processing can delay those adjustments by days. The Discovery team at Shopify uses Apache Beam, a unified batch and stream processing system, to support real-time and historical search-behavior workflows, including boosting, popularity aggregates, offline evaluation sets, and reinforcement learning tasks. The article’s central lesson is that Beam requires careful handling of event time, processing time, delayed events, out-of-order data, and watermarks, especially when combining streams. A Kafka source can use Kafka processing time, client create time with a five-minute out-of-order allowance, or a custom timestamp policy that reads the SearchQueryEvent.searchTimestamp field; Beam windows add further buffering and lateness controls. Accurate timestamping is presented as the first milestone for replaying clickstream data reliably before tackling more complex operations such as streaming joins.


### [How to Reliably Scale Your Data Platform for High Volumes](https://yomu.fyi/post/how-to-reliably-scale-your-data-platform-for-high-volumes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 8, 2020
- Type: Problem & solution

Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.


### [Optimally Scaling Kafka Consumer Applications](https://yomu.fyi/post/optimally-scaling-kafka-consumer-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: Oct 13, 2020
- Type: Problem & solution

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.


### [How to Track State with Type 2 Dimensional Models](https://yomu.fyi/post/how-to-track-state-with-type-2-dimensional-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 27, 2020
- Type: Tutorial

Application databases often retain only current values in Type 1 dimensions, limiting analysis of historical settings such as feature adoption, retention, and switching behavior. The post explains Type 2 dimensional models through Shopify’s need to track users’ admin languages over time, contrasting application-model changes, scheduled database snapshots, and event logging. Its selected implementation uses Rails after\_commit callbacks to send created or updated records to Kafka, then transforms that event history into records with valid\_from, valid\_to, and is\_current fields using ETL recipes involving PySpark and dbt. The approach provides the required granularity but can miss changes or events, requires delete handling and data-quality checks, and is presented as an iterative solution; MySQL binlogs are described as a more reliable future source.


### [Categorizing Products at Scale](https://yomu.fyi/post/categorizing-products-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 30, 2020
- Type: Explainer

Shopify describes a product-categorization system for more than one billion products across the Google Product Taxonomy, which contains over 5,000 hierarchical categories. The model uses product titles, descriptions, collections, tags, vendors, and merchant-provided types, with text preprocessing and a fixed-length term-frequency HashingTF featurizer in PySpark rather than image features or vocabulary-based methods such as Word2Vec. Kesler’s Construction transforms multiclass training into a binary dataset by embedding each candidate class into feature tokens, while logistic regression provides a single, interpretable classifier that can use taxonomy relationships and reduce computational load. The resulting baseline supports more than 20 teams and downstream experiences, while schematized Kafka events and an internal annotation platform provide human feedback; the post identifies class imbalance, multilingual text, and image features as improvement areas.


### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Karan Kamath
- Published: Jan 6, 2020
- Type: Problem & solution

Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.


### [Five Common Data Stores and When to Use Them](https://yomu.fyi/post/five-common-data-stores-and-when-to-use-them.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 15, 2019
- Type: Explainer

The article compares five common data stores—relational databases, NoSQL databases, key-value stores, full-text search engines, and message queues—and connects each to appropriate design needs. Relational databases use schema-bound tables, primary and foreign keys, SQL, and ACID transactions for durable business-critical data, while NoSQL databases trade strict schemas for flexibility with document and wide-column models. In-memory stores such as Redis and Memcached provide fast access and configurable eviction for caching; Redis additionally supports richer data structures and disk persistence. Full-text search engines target word and substring searches, whereas Kafka uses a distributed append-only log and producer-consumer model to move data reliably between services. The article concludes that the right choice depends on whether data must be permanent, flexible, quickly accessed, searched, or transferred.


### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Johan Kok
- Published: Aug 1, 2019
- Type: Problem & solution

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.


### [How Heroku Operates its Multi-Tenant Apache Kafka Services](https://yomu.fyi/post/how-heroku-operates-its-multi-tenant-apache-kafka-services.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ali Hamidi
- Published: Jul 11, 2019
- Type: Explainer

Heroku’s Apache Kafka service supports thousands of developers and millions of transactions, including customers sharing multi-tenant clusters. The post explains how those clusters provide security, predictable performance, availability, and production-like topology at lower cost than single-tenant plans. Multi-tenant and single-tenant offerings use the same numbers of brokers and ZooKeeper nodes, while Kafka ACLs, per-tenant resource namespaces, and producer/consumer byte quotas isolate customers and limit resource contention. Heroku provisions purchased resources, expands storage at 80% of quota, and uses higher defaults for partitions, replicas, and in-sync replicas; exceeding capacity can eventually throttle throughput without stopping the cluster. Automated tests model 50 users with varied traffic, exercise throughput and partition configurations, and simulate broker failures and restarts to identify operational limits and adjust the infrastructure before customer impact.


### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yichao Wang
- Published: Mar 3, 2019
- Type: Problem & solution

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.


[Newer posts](https://yomu.fyi/topic/kafka/page/4.md) · [Older posts](https://yomu.fyi/topic/kafka/page/6.md)