Automated Continuous Deployment at Heroku

Summary

Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.

Takeaways

1. 1

   The age gate keeps a change in staging for 30 minutes before promotion, allowing integration tests to run and potential alerts to appear.

2. 2

   The deployer pauses promotions for open relevant incidents, firing Librato alerts, global or service-specific deployment pauses, and changes that are not present on master.

3. 3

   Automated deployment increased reliance on alarms and platform integration tests, while feature flags and separated database-migration changes helped keep master continuously deployable.