Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku

Summary

The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes.

Takeaways

1. 1

   The three named apps—generate_data, viz, and redshift_batch—actually run as process types within one Heroku app, although they could instead be deployed as separate Heroku apps.

2. 2

   Kafka consumer groups allow redshift_batch to scale horizontally by adding dynos while preserving the immutable event stream for other consumers.

3. 3

   The Redshift consumer commits Kafka offsets after successfully processing database writes, linking offset advancement to completed ingestion.