# Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering

[Grab](https://yomu.fyi/company/grab) · Michael Cartmell · Mar 27, 2019

**Type:** Explainer

## Summary

Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.

## Context

Retries alone only address system recovery and can exacerbate failures if safeguards are missing, while resiliency testing is ineffective unless regularly validated.

## Approach / What changed

Adopt architectural patterns like idempotency keys and asynchronous responses alongside structured, automated chaos engineering experiments on production systems.

## Takeaways

- Idempotency keys allow operations with side-effects to safely retry after timeouts without causing duplicate state changes.
- Asynchronous responses isolate endpoints from dependency failures and latency, but work against the fail-fast principle because background calls might retry indefinitely.
- Chaos experiments require defining a steady state, establishing control and experiment groups, minimizing blast radius, and automating experiment termination if production impact thresholds are exceeded.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://engineering.grab.com/beyond-retries-part-3)