Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)

Summary

Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.

Context

Software services inevitably face upstream failures from network instability, overloads, and resource exhaustion, which can cause cascading latency and crash dependencies if unmanaged.

Approach / What changed

Deploy software circuit breakers such as Hystrix-Go between services to track infrastructure errors, short-circuit traffic when error thresholds are crossed, enforce concurrency limits, and execute fallback routines.

Takeaways

• Circuit breakers should track infrastructure and network errors (e.g., HTTP 500 and 503) rather than user errors (e.g., HTTP 400 and 401) to prevent malicious actors from triggering service disruptions.
• Hystrix-Go includes a bulwark mechanism that bounds concurrent requests, serving as a lightweight rate limiter that prevents local resource starvation and protects upstreams during traffic spikes.
• During open circuit states, services can execute fallbacks—such as using approximate calculations, cached values, or alternative endpoints—or simply fail fast to conserve compute and network resources.

Related reading

• Grab · Jan 8, 2019

  Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)

  Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.

  Corey Scott

• Grab · Mar 20, 2019

  Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting

  Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.

  Michael Cartmell

• Grab · Mar 25, 2019

  Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks

  Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.

  Michael Cartmell

• Grab · Mar 27, 2019

  Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering

  Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.

  Michael Cartmell