---
title: "Modal"
description: "Serverless platform for running compute jobs — AI inference, batch processing, cron, and web endpoints. Pay only for what you use, with custom containers and GPU support."
---

# Modal
> Serverless platform for running compute jobs — AI inference, batch processing, cron, and web endpoints. Pay only for what you use, with custom containers and GPU support.

## Articles

### [Anthropic integration with Modal brings scalable compute to Claude Science](https://yomu.fyi/post/anthropic-integration-with-modal-brings-scalable-compute-to-claude-sci.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 30, 2026
- Type: Announcement

Anthropic launched Claude Science, an AI workbench where life sciences researchers run computational workloads from conversation with Claude. The workbench addresses the fragmented toolchain of local machines, HPC clusters, and cloud VMs by centralizing execution in one place. Modal integrates with Claude Science as the compute backend, routing GPU and CPU-heavy workloads to Modal sandboxes automatically and fanning out parallel tasks like virtual screening across hundreds of containers to reduce hours to minutes. Researchers can prompt for tasks such as predicting GFP structure, and Claude writes the folding code, runs it on a Modal GPU, and returns the structure with confidence scores inside the conversation, with Modal handling scheduling, shared storage, and reproducible environments. Modal is committing up to $100,000 in compute for the Claude Science Cohort, with 500– 2,000 allocations per project.


### [Modal's serverless Servers](https://yomu.fyi/post/modal-s-serverless-servers.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 25, 2026
- Type: Problem & solution

Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.


### [Achieve state-of-the-art inference latencies with speculative decoding](https://yomu.fyi/post/achieve-state-of-the-art-inference-latencies-with-speculative-decoding.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.


### [Introducing Modal Auto Endpoints: Optimized inference you actually own](https://yomu.fyi/post/introducing-modal-auto-endpoints-optimized-inference-you-actually-own.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 23, 2026
- Type: Announcement

Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.


### [Unpacking sandbox startup latency: why started ≠ ready](https://yomu.fyi/post/unpacking-sandbox-startup-latency-why-started-ready.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 22, 2026
- Type: Problem & solution

Sandbox startup benchmarks measure container boot time, but in production application-level initialization after boot dominates, making container boot the smallest variable. Modal defines a lifecycle with events Created, Scheduled, Started, and Ready, where Ready indicates completion of tasks like git pulls or dependency installs. The gap between Started and Ready can be multiple seconds, directly affecting user-perceived latency for coding agents needing cloned repos and running services. To reduce perceived latency, Modal recommends warm pools that pre-initialize sandboxes using a Queue, combined with Directory Snapshots for per-project customization without rebuilding the environment. Readiness Probes, now generally available, allow defining a shell command that exits 0 or a TCP port that accepts connections, with wait\_until\_ready() blocking until the sandbox is truly ready, simplifying warm pool producer code and providing dashboard events for observability.


### [Speculation Is All You Need](https://yomu.fyi/post/speculation-is-all-you-need.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 19, 2026
- Type: Explainer

Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.


### [Product updates: VM Sandboxes, Lower latency routing, RBAC, and more](https://yomu.fyi/post/product-updates-vm-sandboxes-lower-latency-routing-rbac-and-more.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 15, 2026
- Type: Announcement

Modal also announced several product updates including VM Sandboxes in alpha, lower latency regional routing, and role-based access control. VM Sandboxes, currently in alpha, are a new VM runtime that runs a real Linux kernel, supporting Docker containers and Linux features like eBPF, systemd, cgroups, and custom filesystem mounts for better I/O performance. Regional routing lets Functions run in US West, EU West, or Asia Pacific South instead of default US East with a single setting, lowering latency and keeping data in-region. RBAC is now available for Team and Enterprise plans, with Environments as security boundaries and read-only defaults via Restricted Environments, tied into Audit Logs. Other updates include domain allowlisting, named images, volume subdirectory mounts, readiness probes, configurable snapshot TTLs with a 30-day default, SDK updates, and a partnership with Anthropic for Claude Managed Agents.


### [Making FlashAttention-4 faster for inference](https://yomu.fyi/post/making-flashattention-4-faster-for-inference.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jun 11, 2026
- Type: Explainer

FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.


### [Reinforcement learning is an infrastructure problem](https://yomu.fyi/post/reinforcement-learning-is-an-infrastructure-problem.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 1, 2026
- Type: Announcement

Reinforcement learning for post-training large language models has become infrastructure-constrained as teams move to multi-node configurations for larger models like Qwen 3 30B and Kimi K2.6. The RL loop divides into three independent infrastructure challenges: training with reliable forward and backward passes, high-performance inference rollouts, and thousands of concurrent sandbox environments for policy execution. Weight synchronization between trainer and rollout engine becomes the critical bottleneck in multi-node setups, with RDMA offering up to 100x speedup over TCP and significant cost savings per step. Teams commonly get stuck maintaining glue code, queueing for cluster time, and under-utilizing GPUs when sandbox buffers are not correctly sized. Modal addresses these with RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code.


### [Role-Based Access Control for humans and agents](https://yomu.fyi/post/role-based-access-control-for-humans-and-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 27, 2026
- Type: Announcement

Modal announced Role-Based Access Control for Team and Enterprise users, enabling teams to grant agents and humans precise workspace access within Environments. Environments are workspace subdivisions with dedicated storage, apps, and limits, commonly set as dev-staging-production or per-developer. These now act as enforceable security boundaries through Restricted Environments, where workspace members are read-only by default and only explicitly assigned users or service users can deploy or manage resources. The RBAC system distinguishes human developers, AI agents, CI deployers, and external callers with recommended identities and environment access patterns, such as service users for agents and proxy auth tokens for external callers. Built on a new permissions architecture running for months, RBAC ties into Audit Logs and is the first of planned tools; future additions include spending limits and programmatic permissions.


### [Modal's Series C: Raising $355M at a $4.65B valuation](https://yomu.fyi/post/modal-s-series-c-raising-355m-at-a-4-65b-valuation.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 21, 2026
- Type: Announcement

Modal announced a $355 million Series C at a $4.65 billion post-money valuation, led by General Catalyst and Redpoint, with participation from Menlo, Bain Capital Ventures, Accel, and all existing investors. The company grew fivefold since September and surpassed $300 million in annualized revenue. Modal is a cloud platform built for AI workloads, offering elastic inference, dynamic agent runtimes, and batch processing. Its Sandboxes feature, which provides isolated environments for untrusted code, has launched over 1 billion instances and accounts for more than a third of revenue. Modal plans to invest in low-latency inference, reinforcement learning infrastructure, and expanded agent computing capabilities.


### [Scaling reinforcement learning at Applied Compute](https://yomu.fyi/post/scaling-reinforcement-learning-at-applied-compute.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 20, 2026
- Type: Problem & solution

Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.


### [Introducing Claude Managed Agents with Modal Sandboxes](https://yomu.fyi/post/introducing-claude-managed-agents-with-modal-sandboxes.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 19, 2026
- Type: Announcement

Anthropic launched Claude Managed Agents, a suite of composable APIs that separate the agent loop from code execution for building cloud-hosted agents at scale. Modal has collaborated with Anthropic to provide a first-class integration allowing all tool calls to run in self-hosted Modal Sandboxes, which offer fast cold-starts, custom images, snapshotting techniques, and cost-efficient burst pricing. These sandboxes also provide network security through short-lived connect tokens and support upwards of 100,000 concurrent sandboxes per customer. DoorDash is exploring Managed Agents as an accelerator for its merchant intelligence system, while Blend uses the integration to enable secure, scalable agent-assisted triage across complex banking environments. The integration gives teams isolation, cost efficiency, and control over observability, failure handling, and scale while maintaining the convenience of Anthropic’s managed agent loop.


### [How we achieved truly serverless GPUs](https://yomu.fyi/post/how-we-achieved-truly-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: May 12, 2026
- Type: Problem & solution

GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.


### [Boosting multimodal inference performance by >10% with a single Python dictionary](https://yomu.fyi/post/boosting-multimodal-inference-performance-by-10-with-a-single-python-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 4, 2026
- Type: Problem & solution

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.


### [Building an RL theorem-proving workflow on Modal](https://yomu.fyi/post/building-an-rl-theorem-proving-workflow-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Apr 29, 2026
- Type: Explainer

AE Studio built a distributed reinforcement learning pipeline on Modal to train LLMs to prove theorems in Lean. They compared Group Relative Policy Optimization, which groups attempts and applies gradient updates, with Evolution Strategies, which evaluates a population of perturbed models and updates based on success rates. The workflow used Modal’s per-function images to separate a vLLM instance on GPU for proof generation, CPU-based Lean verification, and orchestration, with .map() fanning out runs and Sandboxes isolating verification batches. ES enabled stateless checkpointing via seed histories: each worker loaded base weights from a Volume, replayed past seeds and rewards (about 200 bytes per iteration), and reconstructed the model on GPU without transferring large weight files. Early results on the lean\_workbook dataset using the Kimina-Prover-Preview-Distill-1.5B model showed ES matching or outperforming GRPO in verified proofs per iteration, especially with limited data, though gains varied.


### [Building with Modal and the OpenAI Agents SDK](https://yomu.fyi/post/building-with-modal-and-the-openai-agents-sdk.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Apr 15, 2026
- Type: Tutorial

The post walks through constructing a custom agent harness with the OpenAI Agents SDK and Modal Sandboxes, starting from a minimal coding agent that executes shell commands directly on the host. To improve safety, the agent is moved into isolated Modal Sandboxes, which are secure Linux environments that can be equipped with GPUs for compute-intensive tasks like training models. Memory is added via Sessions to accumulate context across turns, and subagents are introduced to handle focused subtasks with fresh context windows, preventing context bloat in the orchestrator while allowing long-horizon work. An async subagent pool enables parallel execution, with quotas to limit GPU spend, and filesystem snapshots allow checkpointing and deduplication of setup work across subagents, saving time and offloading memory. The harness is applied to the Parameter Golf challenge, where an orchestrator delegates parallel training experiments across different ML frameworks on GPUs, demonstrating autonomous research capabilities with composable skills.


[Newer posts](https://yomu.fyi/company/modal.md)