An enterprise customer with more than 300 Linux and Windows instances in AWS Elastic Disaster Recovery scope had agents stall after network changes and descoped servers keep replicating and adding cost. The solution automates onboarding, healing, and offboarding through AWS Systems Manager Automation documents started by EventBridge or a schedule. A single EC2 DR tag with values yes, pause, or no selects the phase; unrecognized values are normalized, skipped, and logged as SKIPPED_MISCONFIGURED. Onboarding runs every 30 minutes through State Manager, healing restarts or reinstalls the agent for agent-side stalls and escalates infrastructure stalls through Amazon SNS, and offboarding checks safeguards before stopping replication, deleting the source server, uninstalling the agent, and removing tags. Offboarding records whether replication billing stopped, while the documents emit SNS notifications, per-event JSON audit reports to Amazon S3, and CloudWatch Outcome metrics.
Context
An enterprise customer with more than 300 Linux and Windows EC2 instances in AWS DRS scope encountered two problems while planning and validating a scheduled DR drill: some agents had stalled after a network change, and some descoped servers were still replicating and adding to the bill. The team checked replication status in the AWS DRS console before each drill, so a problem that started between drills could go unnoticed until the next one. The automation was built around fixing those two problems.
Approach / What changed
Three SSM Automation documents handle onboarding, healing, and offboarding, with logic in the documents rather than Lambda functions. State Manager runs onboarding every 30 minutes against tag:DR=yes instances; EventBridge triggers healing on STALLED replication events and offboarding on DR tag changes. A DR tag value of yes, pause, or no controls the phase, and unrecognized values are skipped. Healing restarts the service first, then reinstalls in place only if needed, and escalates infrastructure-caused stalls through SNS. Offboarding rechecks the tag, guards against terminated instances, optionally requires approval, records replication state, stops replication, disconnects and deletes the source server, uninstalls the agent, and removes tags.
Takeaways
1
The DR tag value is trimmed and lowercased before evaluation, so DR=Yes matches DR=yes. Any other value, such as DR=maybe or DR=true, is treated as misconfigured: automation skips the instance and logs a SKIPPED_MISCONFIGURED status, preventing an unrecognized value from triggering install, heal, or teardown.
2
Healing inspects the replication error to separate agent-side failures from infrastructure issues. For agent problems it restarts the service first and reinstalls in place only if the restart does not fix it; infrastructure-caused stalls are escalated to operators through Amazon SNS.
3
Offboarding writes one JSON audit object per lifecycle event to Amazon S3, capturing instance ID, host name, platform, source server ID, status, and pre-teardown replication state. CloudWatch Outcome metrics use 1 for handled and 0 for needs attention, dimensioned by event type.
FAQ
What happens when an EC2 instance has an unrecognized DR tag value?
The value is trimmed and lowercased before evaluation. Values other than yes, pause, or no are treated as misconfigured; automation skips the instance and logs a SKIPPED_MISCONFIGURED status, so no install, heal, or teardown runs.
How does healing choose between restarting, reinstalling, and escalating?
It inspects the replication error to separate agent-side failures from infrastructure issues. For agent problems it restarts the service first and reinstalls in place only if the restart does not fix it; infrastructure-caused stalls are escalated to operators through Amazon SNS.
What safeguards does offboarding check before stopping replication?
It performs a tag re-check, a terminated-instance guard, and an optional approval gate. After those pass, it records replication state, stops replication, disconnects and deletes the source server, uninstalls the agent, and removes tags.
Checkpoint/Restore captures live state so work resumes instead of restarting. This architecture view maps the state surface, the capture mechanism, the store tier, and the restore path.
A side-by-side comparison of ad-hoc, human-run remediation and tag-driven AWS Systems Manager Automation, showing what each costs in audit trail, drift, and time to recover.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.