Kafka Streams on Heroku

Summary

Kafka Streams on Heroku shows how to organize, deploy, and scale Java stream-processing services using Kafka Streams, Apache Kafka, and the Heroku Runtime. The example contains a Text Processor that splits input from a textlines topic into words, an Aggregator that counts words in 10-second windows before writing to Postgres, and an independent Anomaly Detector that routes 10-second login-failure counts to urgent alerts or non-urgent email based on thresholds. A Gradle multi-project setup packages each service as an executable and uses a Procfile to run them as separate Heroku workers, while managed Kafka supplies the event-streaming infrastructure. Scaling is explained through Kafka partitions and Kafka Streams Stream Tasks: Stream Threads provide vertical scaling, dyno counts provide independent horizontal scaling, and Kafka Streams redistributes tasks when workers change or fail.

Takeaways

1. 1

   Kafka Streams applications use partitions and Stream Tasks as their fundamental parallelism limits; a service cannot have more Stream Tasks than its topic partitions.

2. 2

   The example Aggregator groups words into 10-second windows and writes counts to Postgres through a custom sink processor, while noting that external-system writes are difficult to implement correctly.

3. 3

   Vertical scaling increases Stream Threads per instance, whereas horizontal scaling changes worker dyno counts so services can be scaled independently.