Circuit Breakers

Shopify · Feb 18, 2020

Your Circuit Breaker is Misconfigured

Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half_open_resource_timeout from 0.25 seconds to 50ms and error_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success_threshold does not affect steady-state utilization.