---
title: "Improved production stability with circuit breakers"
description: "Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures."
---

# Improved production stability with circuit breakers

[Heroku](https://yomu.fyi/company/heroku) · Pedro Belo · Jun 30, 2015

**Type:** Problem & solution

## Summary

Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.

## Context

Heroku's API consumes more endpoints than it serves and depends on over 20 public, internal, and third-party services. The post describes failures including hanging requests, high tail latency, 500s, broken sockets, and bad deploys.

## Approach / What changed

The implementation wraps service calls with Redis-backed CB2 circuit breakers, using individually identified breakers, time windows, percentage-based error thresholds, and fallback handling. The rollout begins in logging-only mode, adds monitoring for state changes, guards against duplicate notifications caused by race conditions, and uses explicit timeouts near network libraries.

## Takeaways

- CB2 uses Redis to maintain circuit-breaker state, and breakers can return cached data or raise a user-friendly exception when a circuit is open.
- Percentage-based thresholds scale better than fixed error counts: making ten times more calls would otherwise make a fixed-count breaker open ten times more often.
- Logging-only rollout, monitored state changes, and locking or similar safeguards help tune breakers and prevent duplicate alerts under heavy traffic.

**Tags:** [Circuit Breakers](https://yomu.fyi/topic/circuit-breakers), [Redis](https://yomu.fyi/topic/redis), [Ruby](https://yomu.fyi/topic/ruby)

- Source: [Heroku](https://www.heroku.com/blog/improved-production-stability-with-circuit-breakers)
- Source URL: https://www.heroku.com/blog/improved-production-stability-with-circuit-breakers
- Ingested by Yomu: 2026-09-09T14:23:58.232Z

[Read original post](https://www.heroku.com/blog/improved-production-stability-with-circuit-breakers)