Building multi-Region resiliency for AWS CloudFormation custom resource deployment

Summary

AWS CloudFormation custom resources extend template provisioning capabilities but lack built-in multi-Region coordination, risking duplicate executions and unhandled failures during regional outages. To solve this limitation, an active-active architecture leverages Amazon Simple Notification Service topics to fan out lifecycle events across primary and secondary infrastructure Regions. The primary Region processes events immediately, whereas the secondary Region processes incoming messages after a deliberate delay implemented via Amazon Simple Queue Service. Coordination relies on Amazon DynamoDB Global Tables with conditional writes to manage distributed locking and enforce idempotency. When failures occur in the primary Region, Amazon CloudWatch alarms signal Amazon Application Recovery Controller to automate failover without manual intervention.

Takeaways

1. 1

   DynamoDB Global Tables use conditional writes to implement distributed locking and idempotency tracking across Regions, preventing race conditions and duplicate side effects.

2. 2

   Local SNS topics in customer Regions use cross-Region subscriptions to simultaneously fan out lifecycle events to SQS queues in primary and secondary central infrastructure Regions.

3. 3

   The secondary infrastructure Region applies an SQS delay queue or visibility timeout to allow the primary Region to process events first, acquiring the lock only during failover scenarios.