Model vs. Harness: Where AI Agent Controls Belong
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
A side-by-side comparison of ad-hoc, human-run remediation and tag-driven AWS Systems Manager Automation, showing what each costs in audit trail, drift, and time to recover.
AWS Systems Manager Automation turns fleet remediation into a declared, repeatable loop instead of a series of hand-run commands. In the closed-loop model, a single EC2 tag selects the phase, EventBridge or a schedule starts the right Automation document, State Manager re-converges the fleet on an interval, and every event emits SNS notifications, JSON audit reports to S3, and CloudWatch outcome metrics. The ad-hoc model is faster to start but leaves no consistent record of what changed, when, or whether it worked, and it depends on someone noticing drift. The comparison matters most at scale, where a stalled agent or a descoped server keeps costing money until something automated finds it, and where destructive steps need safeguards and an audit trail before they run.
The tag value acts as the phase selector: yes, pause, or no picks onboarding, healing, or offboarding. Values that are not recognized are normalized, skipped, and logged as SKIPPED_MISCONFIGURED, so a typo fails visibly instead of silently doing nothing.
Offboarding checks safeguards before it stops replication, deletes the source server, uninstalls the agent, and removes tags. It also records whether replication billing actually stopped, so a server that was descoped but still replicating shows up as a cost problem rather than staying invisible.
A central controller that goes down can halt pipelines and automated remediation at the same time, which is why some platform teams run federated clusters behind GitOps. The loop also depends on agent and network health, since agents can stall after network changes and stop reporting.
The posts behind it
NVIDIA Recovery is an open-source, workload-level service in NVIDIA DSX OS that automates recovery across a cluster’s distributed training jobs. It detects interruptions from application logs, GPU metrics, and scheduler signals, attributes failures to implicated resources, isolates those resources, and coordinates checkpoint restarts on healthy capacity. The recovery loop has four stages: detect crashes, hangs, and stragglers; attribute the interruption; isolate implicated nodes or stop unproductive retries; and restart or stop through scheduler integration. NVIDIA Recovery integrates with Slurm, including Slurm on Kubernetes through slurm-operator, and can optionally use NVIDIA Attribution for deeper diagnosis. In production monitoring from July 9 to October 9, one training cluster covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.
The article describes a structured disaster recovery readiness engagement that compresses the three business decisions and progressive validation framework into a weeks-long process. The kickoff brings together application owner, business owner, and platform team to produce a boundary map listing every service and account in the recovery path with assigned owners and timelines. Assessment uses Well-Architected and Operational Readiness Reviews to surface dependency gaps, which are then addressed with owners and fix dates before testing proceeds. Validation progresses through tabletop exercises, planned failovers, and surprise drills, with each test’s findings fed back into the three business decisions to update the recovery commitment. An enterprise example shows the engagement exposed a missed 5-hour RTO, missing authentication platform, and manual DNS failover, then over four months automated failover with ARC routing controls and pre-provisioned resources to pass the second exercise.
Disaster recovery commitments defined on paper need validation under conditions that expose coordination, dependency, and provisioning gaps. A scheduled failover with advance notice confirms traffic shifts, data consistency in the recovery Region, and application function afterward, but it does not test who declares failover or whether teams coordinate under pressure. The post describes four progressive validation levels: tabletop exercise, planned failover, surprise drill, and game day with fault injection using AWS Fault Injection Service. In one engagement, stopped DR instances depended on launching capacity and database replicas during the exercise without confirming Regional capacity or account quotas; a planned failover found Route 53 records still resolving to the primary Region 45 minutes in, and recovery took more than twice the 5-hour target. Findings map back to the recovery boundary, business-anchored target, and dependency-reconciled commitment, and a later exercise passed.
A disaster recovery strategy executes against targets the business defines, and three decisions made before architecture selection often determine whether teams meet them. The first is drawing the recovery boundary around a business function rather than individual applications, so shared services in other accounts fall inside scope. The second is anchoring recovery time and recovery point objectives to business consequences, with stakeholders confirming the consequence of downtime and the data that must survive. The third is accounting for the full dependency chain, because an application recovers only as fast as its slowest dependency, including services owned by other teams. In one engagement, a shared authentication platform stayed in the primary Region while applications failed over, breaking user login; redefining scope around business functions exposed the dependency and supported a successful test four months later.
An enterprise customer with more than 300 Linux and Windows instances in AWS Elastic Disaster Recovery scope had agents stall after network changes and descoped servers keep replicating and adding cost. The solution automates onboarding, healing, and offboarding through AWS Systems Manager Automation documents started by EventBridge or a schedule. A single EC2 DR tag with values yes, pause, or no selects the phase; unrecognized values are normalized, skipped, and logged as SKIPPED_MISCONFIGURED. Onboarding runs every 30 minutes through State Manager, healing restarts or reinstalls the agent for agent-side stalls and escalates infrastructure stalls through Amazon SNS, and offboarding checks safeguards before stopping replication, deleting the source server, uninstalling the agent, and removing tags. Offboarding records whether replication billing stopped, while the documents emit SNS notifications, per-event JSON audit reports to Amazon S3, and CloudWatch Outcome metrics.
An outage of a central Red Hat Ansible Automation Platform controller halts CI/CD pipelines and automated remediation. To maintain continuous availability, platform teams can deploy a federated multi-cluster architecture managed through GitOps, using Git repositories as a single source of truth for declarative Infrastructure as Code. This approach decouples platform configuration from workload execution, with GitHub Actions coordinating deployment and integrating with ServiceNow for change governance and Splunk for centralized logging. The framework operates in two stages: declarative infrastructure provisioning via the main branch runs playbooks/configure_aap.yml to configure enterprise settings and inventory mappings, while dynamic workload dispatch via the deploy branch uses playbooks/dispatch_round_robin.yml to route jobs across clusters using round-robin or explicit targeting. Key use cases include continuous operations across geographic clusters, rolling updates by diverting workloads to a secondary datacenter, and phased platform migrations by running distinct Ansible Automation Platform versions side by side.
More like this
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
Compare two places to keep Claude's working state: curated files on your own machine versus a durable session store or isolated cloud computer, and learn which one fits your deployment.
CIMD uses an HTTPS URL as an OAuth client identifier that points to a metadata document. This comparison shows when to pre-register clients manually and when to let dynamic discovery handle agent onboarding.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.