---
title: "Kafka"
description: "110 posts about Kafka, summarised, each linking to the original."
---

# Kafka
> 110 posts about Kafka, summarised, each linking to the original.

## Articles

### [Building a Service-oriented Architecture with Rails and Kafka](https://yomu.fyi/post/building-a-service-oriented-architecture-with-rails-and-kafka.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Stella Cotton
- Published: Jan 14, 2019
- Type: Tutorial

Service-oriented Rails applications can use Kafka to move high-volume events between producers and consumers without coupling downstream availability to an upstream request. Kafka persists events for a configured period, lets consumers manage offsets, and distributes replicated data across broker partitions, while ordering is guaranteed within a partition rather than across a topic. The post shows a Rails setup using the ruby-kafka gem, an asynchronous producer, JSON payloads, topics, and partition keys, then applies the event stream to order creation, email, and fulfillment workflows. It also addresses operational concerns including consumer scaling, lag metrics, duplicate delivery, idempotent processing with UPSERTs or event identifiers, and schema evolution, contrasting JSON’s accessibility with Avro’s stronger schema support.


### [Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku](https://yomu.fyi/post/manage-real-time-event-streams-and-sql-analytics-with-apache-kafka-on.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Vikram Rana
- Published: Dec 6, 2018
- Type: Tutorial

The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes.


### [Running Apache Kafka on Kubernetes at Shopify](https://yomu.fyi/post/running-apache-kafka-on-kubernetes-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Sam Obeid
- Published: Nov 7, 2018
- Type: Explainer

Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.


### [Beyond Web and Worker: Evolution of the Modern Web App on Heroku](https://yomu.fyi/post/beyond-web-and-worker-evolution-of-the-modern-web-app-on-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Chris Castle
- Published: Aug 14, 2018
- Type: Explainer

Over a decade, the standard Heroku web application has expanded from a web process, database, worker, and Redis queue into a broader architecture for web, mobile, data, and operational demands. The post connects this shift to snappier single-page experiences, native smartphone apps, APIs as integration surfaces, growing streaming data, increasing system complexity, and scale. Its proposed components include a content delivery network, API Gateway, Apache Kafka, object storage, data warehouses, and query or visualization tools, alongside service-oriented architecture and scalable-first design. The architecture separates functionality into smaller services for adaptability, while acknowledging that more components create more scaling and operational levers. It presents Heroku’s PaaS and managed services as a way to reduce that burden, then leaves deployment, communication, CI/CD, monitoring, and security as topics for follow-up posts.


### [How We Designed the Quotas Microservice to Prevent Resource Abuse](https://yomu.fyi/post/how-we-designed-the-quotas-microservice-to-prevent-resource-abuse.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jim Zhan
- Published: Aug 10, 2018
- Type: Problem & solution

As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.


### [Deconstructing Monolithic Apps with SHIFT Commerce](https://yomu.fyi/post/deconstructing-monolithic-apps-with-shift-commerce.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ryan Townsend
- Published: Mar 13, 2018
- Type: Explainer

SHIFT Commerce describes its move from a monolithic application toward microservices, while arguing that architectural change should address concrete problems rather than follow fashion. The company identifies communication overhead, governance and high-risk deployments, and language or infrastructure limitations as reasons to extract functionality, including its Order Management Suite, admin panel, and integration system. SHIFT uses managed Apache Kafka on Heroku as an asynchronous event stream, with chronologically ordered, durable messages organized into topics so services can maintain local datasets. Its eight-step migration progressively adds producers, builds a consumer-backed database, tests and replicates logic, enables feature-flagged traffic, exchanges events, and removes deprecated monolith code. SHIFT reports more regular releases, lower latency on I/O-restricted API calls, and prevention of user-facing outages, while saying the migration remains ongoing.


### [Kafka Streams on Heroku](https://yomu.fyi/post/kafka-streams-on-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Jeff Chao
- Published: Dec 19, 2017
- Type: Tutorial

Kafka Streams on Heroku shows how to organize, deploy, and scale Java stream-processing services using Kafka Streams, Apache Kafka, and the Heroku Runtime. The example contains a Text Processor that splits input from a textlines topic into words, an Aggregator that counts words in 10-second windows before writing to Postgres, and an independent Anomaly Detector that routes 10-second login-failure counts to urgent alerts or non-urgent email based on thresholds. A Gradle multi-project setup packages each service as an executable and uses a Procfile to run them as separate Heroku workers, while managed Kafka supplies the event-streaming infrastructure. Scaling is explained through Kafka partitions and Kafka Streams Stream Tasks: Stream Threads provide vertical scaling, dyno counts provide independent horizontal scaling, and Kafka Streams redistributes tasks when workers change or fail.


### [Pulling the Thread on Kafka’s Compacted Topics](https://yomu.fyi/post/pulling-the-thread-on-kafka-s-compacted-topics.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Tom Crayford
- Published: Jan 11, 2017
- Type: Incident / postmortem

Apache Kafka’s log cleaner for compacted topics could crash silently when its offset map lacked enough space to hold a full log segment. Because the broker continued running without that internal thread, superseded data accumulated, threatening node and potentially cluster stability. The post explains compaction’s two phases: building an offset map for newer messages, then rewriting segments while retaining the latest offset for each key, with the latest segment excluded from the map. Kafka was changed to remove the crashing assertion and track partial segment loads so cleaning can recover without fitting an entire segment in memory. The roughly 100-line patch shipped in Kafka 0.10.1.1, which the post identifies as Heroku’s default version, and prevents the cleaner from dying silently.


### [Apache Kafka 0.10 Evaluating Performance in Distributed Systems](https://yomu.fyi/post/apache-kafka-0-10-evaluating-performance-in-distributed-systems.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Tom Crayford
- Published: May 27, 2016
- Type: Benchmark

Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.


### [Heroku Metrics](https://yomu.fyi/post/heroku-metrics.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Andrew Gwozdziewycz
- Published: May 26, 2016
- Type: Explainer

Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design.


[Newer posts](https://yomu.fyi/topic/kafka/page/5.md)