Heroku Metrics

Summary

Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design.

Takeaways

1. 1

   Kafka separates ingestion, summarization, and persistence, allowing another datastore to consume summary topics without affecting the existing Postgres store.

2. 2

   The aggregation layer computes count, sum, sum of squares, minimum, and maximum for each measurement over one-minute intervals.

3. 3

   Metrics can still be lost before reaching Kafka because Heroku’s logging pipeline is lossy by design; small losses are typically absorbed by the one-minute rollup strategy.