Improved production stability with circuit breakers

Summary

Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.

Takeaways

1. 1

   CB2 uses Redis to maintain circuit-breaker state, and breakers can return cached data or raise a user-friendly exception when a circuit is open.

2. 2

   Percentage-based thresholds scale better than fixed error counts: making ten times more calls would otherwise make a fixed-count breaker open ten times more often.

3. 3

   Logging-only rollout, monitored state changes, and locking or similar safeguards help tune breakers and prevent duplicate alerts under heavy traffic.