Loading post…
ModalProblem & solution
Scaling reinforcement learning at Applied Compute
Summary
Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.
Context
Applied Compute is a startup founded by former OpenAI researchers that builds custom AI agents for enterprises using reinforcement learning post-training. They use Modal as their cloud infrastructure platform to handle the complex, multi-phase RL training loop.
Approach / What changed
Modal provides ephemeral sandboxes with fast startup for simulating production environments, pre-built cached container images for sub-second cold starts, and serverless fan-out functions for parallel grading. This allows the RL loop to remain GPU-bound and handle thousands of concurrent rollouts and evaluations.
Takeaways
Reinforcement Learning for agent training requires three continuous phases: rollouts in replayable environments, evaluation against reward functions, and inference with trace capture. Each phase has a distinct infrastructure profile—bursty CPU, massively parallel grading, and optimized GPU access.
Applied Compute’s agents are trained in environments that mimic production systems (Salesforce, Slack, internal APIs) to avoid train-test mismatch, a common failure mode in deployed RL. Modal Sandboxes provide the necessary fidelity and determinism for these complex mock environments.
GPU utilization is the dominant cost in RL training; any millisecond of sandbox startup latency translates to idle GPU. Modal’s sub-second cold starts and aggressively cached images keep the training loop GPU-bound, which is essential for serious RL workloads.
FAQ
What makes Applied Compute's approach to AI agents different?
Applied Compute focuses on post-training, where companies own reward functions, evals, and continual learning loops using their proprietary data. This creates specialized agents that improve continuously, unlike static frontier models that set the floor but not the ceiling.
How does Modal help keep GPU utilization high during RL training?
Modal provides sub-second cold starts and pre-built cached container images, so CPU-bound sandbox initialization does not idle the GPU. This keeps the training loop GPU-bound, minimizing wasted accelerator time.
What is the biggest failure mode in deployed RL systems?
Train-test mismatch is the most consistent failure mode. The environment used for training must be indistinguishable from the production environment the agent will encounter. Modal Sandboxes allow constructing arbitrarily complex mocks with necessary fidelity.
Source and freshness
- Source
- modal.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.