---
title: "Amazon ECS for Scheduled Data and ML Work: Five Checkpoints"
description: "Amazon ECS runs containerized batch and ML jobs as scheduled tasks, but the hard parts are capacity, state, and failure behavior. This five-checkpoint path shows what to verify before you scale."
---

# Amazon ECS for Scheduled Data and ML Work: Five Checkpoints

**Type:** [How to](https://yomu.fyi/visual-explainers/category/how-to.md)

![Five numbered checkpoints run left to right across a paper-blue canvas: container readiness, task definition, capacity split by instance type, scheduler cadence, and failure testing, with a lower band noting that state and governance live outside the task.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/df6df231-7195-42dc-a859-d15a9c0fb5cf/image-9b6b6d3f-8e76-4be4-9a7f-b51fb640ee7c.webp)

> Amazon ECS runs containerized batch and ML jobs as scheduled tasks, but the hard parts are capacity, state, and failure behavior. This five-checkpoint path shows what to verify before you scale.

## Explanation

Amazon ECS runs containerized work as tasks, so a data or ML pipeline becomes task definitions plus the capacity that runs them. A practical path confirms the app is genuinely container-ready, splits capacity into separate Auto Scaling groups per instance type, and lets a scheduler like Airflow submit tasks on a cadence. Durable state stays in a database and governance context stays with the data, not inside the ephemeral task. Final checkpoints prove failure behavior with fault injection tests and pin reproducible images so the deployed artifact is verifiable.

## FAQ

### Why split ECS capacity into separate Auto Scaling groups per instance type?

Manta ran Airflow-orchestrated ETL and ML workloads as ECS tasks on separate Auto Scaling groups per instance type, including Graviton-based instances for better price performance. Keeping groups homogeneous lets each family scale on its own signals instead of averaging across incompatible instance shapes. It also makes the cost and performance trade-off of a Graviton move visible per workload.

### How do I know an ECS service will survive an Availability Zone failure?

AWS Resilience Hub's recommended resilience testing generates fault injection experiments from your existing policies, including Availability Zone recovery and dependency validation. Each run returns pass or fail against your success criteria and recovery time objective, plus a point-in-time report. That converts an assumption about failover into evidence you can review.

### Should state and governance live inside the ECS task?

No. ECS tasks are ephemeral, so Manta keeps per-vessel model data in RDS while the task itself is disposable. Governance context can likewise travel with the data rather than the job, as S3 Annotations do when objects are copied or replicated. Keep tasks stateless and let durable stores carry state and policy.

**Tags:** [Amazon ECS](https://yomu.fyi/topic/amazon-ecs.md), [Autoscaling](https://yomu.fyi/topic/autoscaling.md), [AWS](https://yomu.fyi/topic/aws.md), [Containers](https://yomu.fyi/topic/containers.md), [Orchestration](https://yomu.fyi/topic/orchestration.md), [Resilience](https://yomu.fyi/topic/resilience.md)

## Related reading

### [Vessel-specific voyage optimization with machine learning: Manta on AWS](https://yomu.fyi/post/vessel-specific-voyage-optimization-with-machine-learning-manta-on-aws.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Borja Rojo
- Published: Oct 27, 2026
- Type: Problem & solution

Voyage performance varies even between seemingly identical ships due to hull condition, draft, and operational practices, making vessel-type average models inaccurate. Manta Marine Technologies closes this gap by training machine learning models on each vessel’s own operational history combined with weather forecasts. The company built a cloud-native data infrastructure on AWS using Amazon EC2, ECS, and RDS, with Apache Airflow orchestrating ETL and ML workloads as ECS tasks on separate Auto Scaling groups per instance type, including Graviton-based instances for improved price performance. Each vessel’s model is retrained periodically as new voyage data accumulates, creating a continuous learning loop that refines voyage guidance. Based on internal simulations, Manta expects fuel savings of approximately 2–4 percent per vessel, with further gains anticipated as models improve.


### [How Anchorage Digital and Turnkey verify code with AWS Nitro Enclaves](https://yomu.fyi/post/how-anchorage-digital-and-turnkey-verify-code-with-aws-nitro-enclaves.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Alexander Qin
- Published: Oct 27, 2026
- Type: Problem & solution

Cryptocurrency wallets face phishing attacks due to blind signing, where transactions are approved without full semantic understanding. Anchorage Digital and Turnkey built VisualSign, an open source Rust transaction parser that runs inside AWS Nitro Enclaves. The enclave communicates only over a local vsock channel with no persistent storage or network, and produces signed attestation documents containing Platform Configuration Registers (PCRs) that prove which exact code executed. Reproducible builds ensure byte-identical binaries, so PCR values are deterministic and verification policies never require updates. This architecture pattern of reproducible builds, enclave-hosted computation, and attestation-gated access extends beyond blockchain as a general-purpose verifiable compute building block.


### [Deploying third-party applications to Red Hat OpenShift: Common pitfalls and how to avoid them](https://yomu.fyi/post/deploying-third-party-applications-to-red-hat-openshift-common-pitfall.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Alasdair Preston
- Published: Oct 27, 2026
- Type: Explainer

Enterprise customers deploying third-party applications on Red Hat OpenShift often encounter vendors claiming Kubernetes support without addressing the underlying architectural changes needed for containerization. These applications are frequently repackaged from legacy virtual machines, retain tight pod interdependencies, and demand cluster-admin privileges or unrestricted security context constraints, violating OpenShift’s default restricted-v2 SCC. Many products fail to support parallel dev, test, and staging environments because they depend on cluster-scoped resources, and they rely on legacy NFS shared storage instead of dynamic provisioning via OpenShift Data Foundation. Avoiding these pitfalls requires formal architectural reviews with vendor engineering, strict enforcement of non-root execution within defined SCC policies, and conversion of manual vendor scripts into declarative Helm charts managed through GitOps. Consultants must also manage expectations with evidence-based feedback and ensure thorough knowledge transfer so the customer’s own engineering team can maintain a clean, repeatable platform.


### [Data governance with annotations in Amazon S3](https://yomu.fyi/post/data-governance-with-annotations-in-amazon-s3.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Aditya Jain
- Published: Oct 26, 2026
- Type: Tutorial

Amazon S3 Annotations lets you attach governance context directly to S3 objects without rewriting them. Each object version can hold up to 1,000 annotations, each 1 byte to 1 MB in UTF-8 format such as JSON. Annotations travel with the object during copy or replication, remain readable in cold storage like Glacier Deep Archive without restoration, and can be updated in place without changing the object’s ETag. The pattern uses a governance annotation payload capturing classification, retention, jurisdiction, PII, and rationale, and supports backfill of existing objects. At scale, an S3 Metadata annotation table backed by Apache Iceberg enables SQL querying via Amazon Athena for governance questions like which objects hold PII and have expired retention across large object populations.


### [Recommended resilience testing in the next generation of AWS Resilience Hub](https://yomu.fyi/post/recommended-resilience-testing-in-the-next-generation-of-aws-resilienc.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Jennifer Moran
- Published: Oct 26, 2026
- Type: Announcement

AWS Resilience Hub introduced recommended resilience testing, which uses AWS Fault Injection Service to automatically create fault injection experiments from existing policies and resource knowledge. The feature offers four pre-configured tests: Availability Zone recovery, dependency validation, Multi-Region isolation, and Multi-Region recovery. Dependency validation blocks connectivity to selected dependencies using DNS query log analysis with a 35-day look-back window, and validates whether the application remains healthy when they are unavailable, surfacing misclassified dependencies. Each test run produces a pass or fail outcome evaluated against success criteria and the recovery time objective defined in your resilience policy, with an auto-generated report capturing a point-in-time snapshot of resources. This capability removes manual experiment design, helps organizations prove their architecture holds under failure, and feeds results into compliance and reporting workflows.
