# Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting

[Grab](https://yomu.fyi/company/grab) · Michael Cartmell · Mar 20, 2019

**Type:** Explainer

## Summary

Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.

## Context

At scale, reliance on retries risks triggering retry storms, while circuit breakers depend on error-prone client-side configurations that servers cannot safely trust to prevent service overloads.

## Approach / What changed

Deploying layered rate limiting across per-client, per-endpoint, and server-wide thresholds, backed by centralized global rate limiters like Grab's Quotas service alongside optional client-side quota headers.

## Takeaways

- Local instance-level rate limiting is ineffective against downstream resource exhaustion because horizontal scaling increases total aggregate pressure on databases and dependent services.
- Layering rate limits across per-client-per-endpoint, per-client, per-endpoint, and server-wide rules captures individual SLA adherence while defending against overall server overhead.
- Centralized rate-limiting systems must process metrics asynchronously or support local fallbacks to avoid becoming single points of failure and adding latency to the request path.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/beyond-retries-part-1)