---
title: "Heroku Metrics"
description: "Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design."
---

# Heroku Metrics

[Heroku](https://yomu.fyi/company/heroku) · Andrew Gwozdziewycz · May 26, 2016

**Type:** Explainer

## Summary

Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design.

## Context

The original metrics infrastructure was not robust to node restarts, crashes, or other failures, and adding one-minute resolution doubled CPU load through InfluxDB continuous queries. The team also wanted a system that could support richer operational experiences, remain extensible and robust, and run on Heroku.

## Approach / What changed

A Kafka-based stream-processing pipeline runs in Private Spaces. Metrics ingress converts Logplex frames into Protocol Buffers messages, Go-based aggregation consumers compute one-minute rollups, and a sink persists summarized data in seven sharded and partitioned Heroku Postgres databases. The custom Yahtzee framework uses deterministic partition assignment and accumulator maps.

## Takeaways

- Kafka separates ingestion, summarization, and persistence, allowing another datastore to consume summary topics without affecting the existing Postgres store.
- The aggregation layer computes count, sum, sum of squares, minimum, and maximum for each measurement over one-minute intervals.
- Metrics can still be lost before reaching Kafka because Heroku’s logging pipeline is lossy by design; small losses are typically absorbed by the one-minute rollup strategy.

**Tags:** [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Postgres](https://yomu.fyi/topic/postgres), [Protocol Buffers](https://yomu.fyi/topic/protobuf)

- Source: [Heroku](https://www.heroku.com/blog/heroku-metrics-there-and-back-again)
- Source URL: https://www.heroku.com/blog/heroku-metrics-there-and-back-again
- Ingested by Yomu: 2026-09-09T14:24:42.232Z

[Read original post](https://www.heroku.com/blog/heroku-metrics-there-and-back-again)