Validating multi-Region DR for Terraform Enterprise with AWS FIS

Summary

Athenahealth’s single-Region Terraform Enterprise deployment became inaccessible to developers during an October 2025 AWS regional service event, exposing missing regional failover capabilities. The post describes a customer-operated active-passive design spanning us-east-1 and us-west-2, using Aurora PostgreSQL-Compatible global databases, bidirectional S3 cross-Region replication, Route 53 health checks, and compute scaled to zero in the DR Region. AWS Fault Injection Service experiments validate three phases of recovery, including failover, failback, and disruptions that can reveal dependencies such as state-file lookups or S3 connectivity. The resulting architecture achieved a validated 12–14 minute RTO and less than one minute RPO, while costing approximately 30–40% more than the single-Region deployment. The design is presented as an option for critical workloads rather than a supported multi-Region TFE product configuration.

Takeaways

1. 1

   Aurora PostgreSQL-Compatible global databases provide sub-second replication lag, while bidirectional S3 replication supports failback without data resynchronization.

2. 2

   The TFE encryption password must be replicated and referenced in the DR launch configuration; a different value can prevent instances from starting or decrypting existing data.

3. 3

   Subnet-level AWS FIS connectivity disruption can simulate regional S3 connectivity loss and expose circular dependencies involving Terraform state files.