Model vs. Harness: Where AI Agent Controls Belong
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
ModalKevin YangExplainer
Modal Sandboxes are bursty workloads that spend most of their time using far less than peak CPU, making reservation-based pricing wasteful. The post argues that reserving capacity for each Sandbox’s peak would require machine capacity equal to the sum of all peaks, while bursting allows multiple Sandboxes to share capacity efficiently. Modal uses active-resource pricing where Sandboxes are billed for a default request when idle and pay for additional resources only when they burst. CPU can burst up to 16 cores above the request, and memory has no limit by default, though limits can be set for consistency or cost capping. Evictions are rare, with an SLO of under 0.0001% OOM rate per minute for Sandboxes under 8 GiB, and tools like exit snapshots and volumes help mitigate their impact.
Modal Sandboxes are compute environments used for workloads such as coding agents, RL rollouts, evals, and app builders. These workloads are bursty: a typical Sandbox spends more than half its life using less than 6% of its peak CPU, with spikes only during tasks like installing dependencies or running tests. The post argues that reserving capacity for peak demand is wasteful, and that bursting allows multiple Sandboxes to share machine capacity efficiently.
Modal uses active-resource pricing where Sandboxes are billed for a default request when idle and pay for additional resources only when they burst above that request. CPU can burst up to 16 cores above the request, and memory has no limit by default. A limit can be set for consistency or cost capping, and memory limits stop Sandboxes that cross them. Exit snapshots, filesystem snapshots, volumes, and the termination reasons SDK help reduce the impact of evictions.
A typical Modal Sandbox spends more than half its life using less than 6% of its peak CPU, with spikes only during tasks like installing dependencies or running tests.
With bursting, multiple Sandboxes on one machine share capacity efficiently because their spikes rarely align, so the machine only needs room for the most concurrent use, not the sum of all peaks.
Modal guarantees at least the requested CPU to each Sandbox, and memory evictions are rare: for Sandboxes under 8 GiB, the OOM rate is below 0.0001% per minute (about 1 eviction per 694 Sandbox-days).
By default, CPU can burst up to 16 cores above the request, and memory has no limit. A limit is optional and can be set for consistency in benchmarks or to cap costs. If a memory limit is set, Sandboxes that cross it are stopped.
If a machine runs out of CPU, bursting Sandboxes slow down until CPU frees up. If a machine runs out of memory, Sandboxes using more than their request at that moment may be evicted with an out-of-memory error. Sandboxes within their request are never evicted.
For bursting Sandboxes under 8 GiB, Modal provides an SLO of an OOM rate below 0.0001% per minute, which is about 1 eviction per 694 Sandbox-days. There is no SLO yet for larger Sandboxes, but they also see low eviction rates in practice.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.