---
title: "Validating your DR commitment: From controlled test to real-world readiness"
description: "Disaster recovery commitments defined on paper need validation under conditions that expose coordination, dependency, and provisioning gaps. A scheduled failover with advance notice confirms traffic shifts, data consistency in the recovery Region, and application function afterward, but it does not test who declares failover or whether teams coordinate under pressure. The post describes four progressive validation levels: tabletop exercise, planned failover, surprise drill, and game day with fault injection using AWS Fault Injection Service. In one engagement, stopped DR instances depended on launching capacity and database replicas during the exercise without confirming Regional capacity or account quotas; a planned failover found Route 53 records still resolving to the primary Region 45 minutes in, and recovery took more than twice the 5-hour target. Findings map back to the recovery boundary, business-anchored target, and dependency-reconciled commitment, and a later exercise passed."
---

# Validating your DR commitment: From controlled test to real-world readiness

[AWS](https://yomu.fyi/company/aws) · Peter Garnepudi · Nov 2, 2026

**Type:** Explainer

## Summary

Disaster recovery commitments defined on paper need validation under conditions that expose coordination, dependency, and provisioning gaps. A scheduled failover with advance notice confirms traffic shifts, data consistency in the recovery Region, and application function afterward, but it does not test who declares failover or whether teams coordinate under pressure. The post describes four progressive validation levels: tabletop exercise, planned failover, surprise drill, and game day with fault injection using AWS Fault Injection Service. In one engagement, stopped DR instances depended on launching capacity and database replicas during the exercise without confirming Regional capacity or account quotas; a planned failover found Route 53 records still resolving to the primary Region 45 minutes in, and recovery took more than twice the 5-hour target. Findings map back to the recovery boundary, business-anchored target, and dependency-reconciled commitment, and a later exercise passed.

## Context

Part 2 of a three-part series. Part 1 defined three business decisions shaping a disaster recovery commitment: the recovery boundary, the business-anchored target, and the dependency-reconciled commitment. This part treats testing as a progressive practice that surfaces different gap classes and feeds findings back into those decisions.

## Approach / What changed

Use four progressive validation levels ordered from low-cost awareness to high-fidelity validation: a tabletop walkthrough, a planned failover with advance notice, a surprise drill without notice, and a game day with fault injection using AWS Fault Injection Service. At each level, measure actual recovery time against RTO, validate RPO through data consistency, identify manual steps and late dependencies, and map findings to the recovery boundary, business-anchored target, or dependency-reconciled commitment. Pre-provision the recovery environment, confirm account quotas and Regional capacity, and remove runbook steps that assume resource creation during the event.

## Takeaways

- A scheduled failover with advance notice can confirm the failover mechanism, traffic shifting, data consistency, and application function, but it cannot test coordination among unrehearsed teams, failover declaration authority, or whether the recovery environment was pre-provisioned.
- The four validation levels are tabletop exercise, planned failover, surprise drill, and game day with fault injection; each surfaces a different class of gap, from unowned steps to misconfigurations, coordination failures, and unmodeled dependencies.
- In the case study, a planned failover found Route 53 records still resolving to the primary Region 45 minutes in and recovery took more than twice the 5-hour target; a surprise drill cost 90 minutes when only one engineer knew how to validate database consistency in the recovery Region.

## FAQ

### What does a tabletop exercise reveal that a planned failover does not?

A tabletop walkthrough surfaces ownership and boundary gaps without touching production. In the case study, it found that the shared authentication platform was outside recovery scope, a networking account had no failover runbook, and the DNS update step named a generic team alias rather than an owner.

### Why can a scheduled failover miss real recovery failures?

Advance notice and close observation let every team prepare, so the exercise proves the mechanism under known conditions. It does not show whether teams can coordinate under pressure, who declares failover, or whether recovery capacity and quotas were confirmed before the event.

### How do findings from each test feed back into the DR commitment?

A boundary gap feeds back into the recovery boundary decision, a recovery that exceeds business tolerance feeds back into the business-anchored target, and a dependency that recovers slower than assumed feeds back into the dependency-reconciled commitment. The team then maps each finding to a specific fix.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Disaster Recovery](https://yomu.fyi/topic/disaster-recovery), [Fault Injection](https://yomu.fyi/topic/fault-injection), [Resilience](https://yomu.fyi/topic/resilience)

- Source: [AWS](https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness)
- Source URL: https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness
- Ingested by Yomu: 2026-11-02T18:01:10.389Z

[Read original post](https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness)
