Your Circuit Breaker is Misconfigured

Summary

Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half_open_resource_timeout from 0.25 seconds to 50ms and error_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success_threshold does not affect steady-state utilization.

Context

Dependency timeouts can keep workers blocked on I/O, causing request queues and utilization to rise until the application is effectively down. Circuit-breaker settings also need to account for multiple failing service instances, false-positive openings, partial outages, recovery behavior, and the capacity available to a worker.

Approach / What changed

The post analyzes Semian’s name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold parameters, models their effect with a circuit-breaker equation, and validates that equation with a live test. It then tunes the settings for a two-thread Rails worker handling 42 Redis instances.

Takeaways

• Each service instance should have its own circuit-breaker name so one failing Redis instance does not open circuits for other instances or the entire service type.
• Reducing error_threshold limits the initial utilization spike, but increases the chance that brief service blips will trigger false-positive circuit openings.
• For the example Rails worker, lowering half_open_resource_timeout to 50ms and increasing error_timeout to 30 seconds reduced modeled extra utilization from 263% to 4%, at the cost of slower recovery.

Related reading