---
title: "Circuit Breakers: The Recovery Loop That Spends Capacity"
description: "A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity."
---

# Circuit Breakers: The Recovery Loop That Spends Capacity

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![A top-to-bottom four-stage diagram shows a circuit breaker moving from Closed error counting to Open fast-fail, then Half-Open limited probing, then back to Closed or Open; a side callout shows a half-open probe occupying worker capacity.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/15903ba9-f85c-4bec-9774-195e4a17202b/image-1036e320-1b66-4fbe-a0c4-b87bd5e51894.webp)

> A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.

## Explanation

A circuit breaker wraps calls to a dependency and watches errors over a time window. When the error rate crosses a threshold, it opens and returns a fallback or fast failure instead of letting slow calls pile up. After a cooldown it moves to half-open, allowing limited probes; enough successes close it, while a failed probe reopens it. The practical trap is that half-open probes still occupy worker capacity, so timeout and cooldown settings must be tuned together rather than treated as independent knobs.

## FAQ

### Why can a misconfigured circuit breaker reduce availability instead of protecting it?

When a breaker opens, half-open probes can still be allowed to run until they time out. On a small worker pool those timed-out probes occupy capacity, so the dependency outage becomes a local capacity outage; tuning probe timeout and cooldown together limits that waste.

### Should the error threshold be a count or a percentage?

A fixed error count does not scale cleanly as traffic or instance count grows. Heroku’s CB2 uses percentage-based thresholds so the breaker trips on error rate rather than an absolute number; either way, align the threshold with the error window you actually observe.

### How should a team roll out its first circuit breakers?

Start in logging-only mode so parameters and unexpected internal-API behavior can be reviewed before traffic is affected. Then monitor state changes, guard against duplicate notifications from race conditions, and set explicit low network-level timeouts so slow dependencies fail fast.

**Tags:** [Backpressure](https://yomu.fyi/topic/backpressure.md), [Circuit Breakers](https://yomu.fyi/topic/circuit-breakers.md), [Latency](https://yomu.fyi/topic/latency.md), [Monitoring](https://yomu.fyi/topic/monitoring.md), [Reliability](https://yomu.fyi/topic/reliability.md), [Resilience](https://yomu.fyi/topic/resilience.md)

## Related reading

### [Well, That Wasn't Fun: October 16 '26](https://yomu.fyi/post/well-that-wasn-t-fun-october-16-26.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 6, 2026
- Type: Incident / postmortem

Fly.io responded to CVE-2026-99999, a Linux kernel bug in IPv4 checksum-offload metadata validation, by patching and rebooting its entire fleet over 10 hours on October 16, 2026. The bug allowed a guest to rewrite two bytes of an IPv4 header after the host applied routing and firewall rules, enabling firewall bypass and an out-of-bounds read of up to 32 bytes of host memory. The team split each region into waves so no Managed Postgres cluster had both nodes rebooted simultaneously, with the largest region requiring 17 waves. Some hosts failed to return cleanly, including a chassis in gru that lost its power supply, delaying recovery for those machines until October 20. The post-mortem led to a new Incident Communications Plan that shifts communication decisions from incident responders to the exec team.


### [A builder’s guide to scaling Databricks Apps](https://yomu.fyi/post/a-builder-s-guide-to-scaling-databricks-apps.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Bruno Tafur, Tushar Madan
- Published: Nov 5, 2026
- Type: Tutorial

As a Databricks Apps application scales from a few users to thousands across an organization, it must handle growing demand without latency spikes or outages. The guide proposes a four-stage approach: set targets during design, build with async code and recoverable state, test with load simulations, and operate with monitoring. Keep app compute focused on serving the UI; send heavy work to Lakebase, Databricks Jobs, or Sandbox, and choose a framework like AppKit that offers built-in caching and telemetry. Separate session state into soft state (rebuildable from hard state) and hard state stored durably in Lakebase or Unity Catalog so any instance can reconstruct a session after restarts or scaling. Use load tests to validate concurrency and bursts, then consult system metrics on CPU, memory, timeouts, and connections to decide whether to scale up to a larger instance or scale out with more instances.


### [Your Circuit Breaker is Misconfigured](https://yomu.fyi/post/your-circuit-breaker-is-misconfigured.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 18, 2020
- Type: Explainer

Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error\_threshold, error\_timeout, half\_open\_resource\_timeout, and success\_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half\_open\_resource\_timeout from 0.25 seconds to 50ms and error\_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success\_threshold does not affect steady-state utilization.


### [Improved production stability with circuit breakers](https://yomu.fyi/post/improved-production-stability-with-circuit-breakers.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Pedro Belo
- Published: Jun 30, 2015
- Type: Problem & solution

Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
