---
title: "Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku"
description: "The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes."
---

# Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku

[Heroku](https://yomu.fyi/company/heroku) · Vikram Rana · Dec 6, 2018

**Type:** Tutorial

## Summary

The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes.

## Context

The system addresses the need to manage high-volume event streams for near-real-time product-usage monitoring while also moving those events into a data warehouse for deeper questions, trend analysis, dashboards, and ad-hoc SQL queries.

## Approach / What changed

A Node.js producer publishes events to Heroku-managed Apache Kafka. Separate consumers visualize selected events through an Express and WebSocket application and batch Kafka events into Amazon Redshift using consumer groups and offset commits. Metabase provides SQL analytics and dashboards on the warehouse data.

## Takeaways

- The three named apps—generate\_data, viz, and redshift\_batch—actually run as process types within one Heroku app, although they could instead be deployed as separate Heroku apps.
- Kafka consumer groups allow redshift\_batch to scale horizontally by adding dynos while preserving the immutable event stream for other consumers.
- The Redshift consumer commits Kafka offsets after successfully processing database writes, linking offset advancement to completed ingestion.

**Tags:** [Heroku](https://yomu.fyi/topic/heroku), [Kafka](https://yomu.fyi/topic/kafka), [Node.js](https://yomu.fyi/topic/node-js), [SQL](https://yomu.fyi/topic/sql)

- Source: [Heroku](https://www.heroku.com/blog/event-streams-kafka-redshift-metabase)
- Source URL: https://www.heroku.com/blog/event-streams-kafka-redshift-metabase
- Ingested by Yomu: 2026-09-09T14:26:54.032Z

[Read original post](https://www.heroku.com/blog/event-streams-kafka-redshift-metabase)