Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)

Summary

Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.

Takeaways

1. 1

   Circuit breakers should track infrastructure and network errors (e.g., HTTP 500 and 503) rather than user errors (e.g., HTTP 400 and 401) to prevent malicious actors from triggering service disruptions.

2. 2

   Hystrix-Go includes a bulwark mechanism that bounds concurrent requests, serving as a lightweight rate limiter that prevents local resource starvation and protects upstreams during traffic spikes.

3. 3

   During open circuit states, services can execute fallbacks—such as using approximate calculations, cached values, or alternative endpoints—or simply fail fast to conserve compute and network resources.