Evolving the Backend Storage for Platform Metrics

Summary

Heroku migrated MetaaS, its internal Metrics as a Service platform metrics backend, from self-hosted Apache Cassandra to AWS’s DynamoDB. MetaaS collects application observations, aggregates them into per-application, per-minute statistics such as median, maximum, and 99th-percentile response time, and stores time-series data for dashboard metrics, alerting, and autoscaling. The migration first duplicated Kafka stream-processing writes into DynamoDB, then used “Science Mode” to compare Cassandra and DynamoDB read results in production without changing returned data. After edge cases were fixed and results matched, retention-period TTL expiry avoided a separate historical data transfer, while gradual query cutovers provided another safeguard. Since 100% of MetaaS queries moved to DynamoDB in May of last year, DynamoDB has remained reliable at scale and been faster and cheaper than the Cassandra clusters, with no noticeable effect on users.

Takeaways

1. 1

   MetaaS processes a couple hundred thousand observations per second and stores many terabytes of time-series data, with tens of thousands of writes and several thousand reads per second at peak.

2. 2

   Science Mode queried DynamoDB in the background while Cassandra continued serving responses, exposing edge cases before the new backend became authoritative.

3. 3

   TTL-based retention meant older Cassandra data could age out naturally, avoiding a lift-and-shift; DynamoDB was later reported to be faster, cheaper, and reliable at scale.