# Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks

[Grab](https://yomu.fyi/company/grab) · Michael Cartmell · Mar 25, 2019

**Type:** Explainer

## Summary

Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.

## Context

Retrying failed API calls is insufficient when a remote server is completely down or returning continuous errors, risking cascading system failures.

## Approach / What changed

Implement bulkheading at multiple levels, distribute traffic across backend server pools using various load balancing architectures, and apply graceful degradation fallback strategies.

## Takeaways

- Bulkheading can be enforced at multiple architectural layers, from cloud regions and operating systems down to distinct processes, connection pools, and thread limits.
- Load balancing architectures span traditional proxies, client-side implementations, lookaside services, and sidecar service meshes, each carrying distinct network hop and maintenance trade-offs.
- Fallback mechanisms for unrecoverable failures include failing silently with null responses, local computations, dual-expiration fallback caching, and isolated backup services.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Caching](https://yomu.fyi/topic/caching), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/beyond-retries-part-2)