# Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)

[Grab](https://yomu.fyi/company/grab) · Corey Scott · Dec 21, 2018

**Type:** Explainer

## Summary

Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.

## Context

Software services inevitably face upstream failures from network instability, overloads, and resource exhaustion, which can cause cascading latency and crash dependencies if unmanaged.

## Approach / What changed

Deploy software circuit breakers such as Hystrix-Go between services to track infrastructure errors, short-circuit traffic when error thresholds are crossed, enforce concurrency limits, and execute fallback routines.

## Takeaways

- Circuit breakers should track infrastructure and network errors (e.g., HTTP 500 and 503) rather than user errors (e.g., HTTP 400 and 401) to prevent malicious actors from triggering service disruptions.
- Hystrix-Go includes a bulwark mechanism that bounds concurrent requests, serving as a lightweight rate limiter that prevents local resource starvation and protects upstreams during traffic spikes.
- During open circuit states, services can execute fallbacks—such as using approximate calculations, cached values, or alternative endpoints—or simply fail fast to conserve compute and network resources.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Go](https://yomu.fyi/topic/go), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/designing-resilient-systems-part-1)