# Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)

[Grab](https://yomu.fyi/company/grab) · Corey Scott · Jan 8, 2019

**Type:** Explainer

## Summary

Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.

## Context

Managing request failures in distributed systems requires evaluating retry strategies alongside circuit breakers, particularly addressing how to handle partial failures, host outages, and load spikes.

## Approach / What changed

Implement retries using error filtering, exponential backoff, randomized jitter, cryptographic nonces for idempotency, and carefully bounded maximum retry and delay settings.

## Takeaways

- Errors indicating potential transient recovery (HTTP 500 and 503) should be retried, whereas client-side errors (HTTP 400 and 401) must not be retried.
- Non-idempotent operations can use a cryptographic nonce attached to the request payload to allow upstream hosts to detect and coalesce duplicate transactions.
- Worst-case consumer response time can be calculated as the product of maximum retries and request timeout plus the product of maximum retries and maximum delay.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Reliability](https://yomu.fyi/topic/reliability), [REST APIs](https://yomu.fyi/topic/rest-api), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/designing-resilient-systems-part-2)