---
title: "Modal"
description: "Serverless platform for running compute jobs — AI inference, batch processing, cron, and web endpoints. Pay only for what you use, with custom containers and GPU support."
---

# Modal
> Serverless platform for running compute jobs — AI inference, batch processing, cron, and web endpoints. Pay only for what you use, with custom containers and GPU support.

## Articles

### [Modal crosses $1B in Annualized Run Rate and raises $750M Series D](https://yomu.fyi/post/modal-crosses-1b-in-annualized-run-rate-and-raises-750m-series-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Erik Bernhardsson
- Published: Oct 14, 2026
- Type: Announcement

Modal crossed $1B in annualized run rate and raised $750M in Series D funding at a $15.75B post-money valuation, led by Accel. Usage has tripled since the Series C five months ago, with year-over-year growth exceeding 15x, and the company now powers applications from companies including DoorDash, Cognition, and Runway. To avoid the friction of the old stack, Modal built its own file system, container runtime, scheduler, and networking from scratch on a capacity pool spanning over 20 clouds. Sandboxes, launched in 2023 for running LLM-generated code, now represent a $300M run rate product with nearly 3 billion sandboxes spun up and scaling to a million concurrent. New products include multi-node training with a single decorator giving a fully RDMA-connected cluster, Modal servers for HTTP serving, Endpoint Candidates for LLM serving, and generally available virtual machines.


### [Endpoint Candidates: evaluate and promote inference endpoints](https://yomu.fyi/post/endpoint-candidates-evaluate-and-promote-inference-endpoints.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Amit Lohe, Deven Navani, Garrett O'Brien
- Published: Oct 12, 2026
- Type: Announcement

Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.


### [TypeSafe AI scales Jev to trillions of tokens in three days on Modal](https://yomu.fyi/post/typesafe-ai-scales-jev-to-trillions-of-tokens-in-three-days-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Oct 5, 2026
- Type: Announcement

TypeSafe AI’s Jev is a System One Model that outputs structured decisions with probabilities rather than generating text, taking state and questions as input and returning typed values. It uses a custom architecture, a parallel sampler for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Unlike autoregressive LLMs, Jev produces outputs in parallel, enabling low-latency responses suitable for workloads like large-scale automation, fast browser use, and generative user interfaces. Launched on a Thursday, Jev topped OpenRouter’s leaderboard for contexts between 1K and 10K tokens one week later, handling 15.1% of requests on that platform. To meet demand exceeding one trillion tokens in three days, TypeSafe used Modal’s serverless platform for bursty, high-concurrency inference without managing their own infrastructure, gaining control over networking, routing, and autoscaling needed for ultra-low-latency serving.


### [Modal Clusters are generally available](https://yomu.fyi/post/modal-clusters-are-generally-available.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Peyton Walters
- Published: Oct 1, 2026
- Type: Announcement

Modal announced general availability of Modal Clusters, a serverless multi-node compute primitive for AI workloads that provides up to 6.4 Tbps InfiniBand RDMA per node with PyTorch and NCCL support. A new gang scheduler places N nodes simultaneously by observing the fleet, assigning all pending clusters, and provisioning capacity when needed. RDMA support is fully automated behind a single rdma=True flag, with Modal contributing RDMA support to the gVisor container runtime and upstreaming the changes. Customers like Decagon, 1x, and Runway use Clusters for trillion-parameter fine-tuning, pre-training home robot world models, and multi-node video inference respectively. Clusters are available to all workspaces today through the @modal.clustered decorator, with per-second billing and no reservation required.


### [VM Sandboxes: Full computers for agents](https://yomu.fyi/post/vm-sandboxes-full-computers-for-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Oct 1, 2026
- Type: Announcement

Modal has launched VM Sandboxes in general availability, providing agents with full Linux VMs accessible via a single flag. The VMs support Docker stacks, local databases, dev servers, and other real-machine features that agents increasingly require, as demonstrated by Ramp’s internal agent driving the entire software development lifecycle. Modal built a custom runtime on the Rust-based Cloud Hypervisor, layering innovations in host filesystems, lazy image loading, memory bursting, and snapshotting to preserve the same APIs and sub-second cold starts as their existing gVisor Sandboxes. Early customers like Linear, Legora, and Snorkel have launched over 20 million VMs, using them for coding sessions, legal agent evaluations, and agent simulations that require multiple containers and real-world environments. Modal recommends using VM Sandboxes when workloads need Docker, FUSE filesystems, or niche kernel features, while gVisor remains the default for compatible workloads.


### [Runtime Roundup: VM Sandboxes, Multi-node clusters, and more](https://yomu.fyi/post/runtime-roundup-vm-sandboxes-multi-node-clusters-and-more.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Oct 1, 2026
- Type: Announcement

Modal announced several new features at its inaugural Runtime conference. VM Sandboxes provide agents with full Linux computer access, supporting Docker stacks, databases, graphical environments, and kernel experimentation, already used at Linear, Legora, and Snorkel for coding agents and evals. Sandbox Sidecars run alongside main sandboxes with isolation but share a host, allowing agent code in the main sandbox while trusted operations run in a sidecar the agent cannot access, avoiding network round trips in execution-heavy loops. Modal Clusters, now generally available via @modal.clustered, enable multi-node jobs for training or serving trillion-parameter models, powering post-training at Decagon and real-time inference at Runway. Sticky Sessions maintain client state by routing to the same container using a session token, critical for voice model serving, while Endpoint Candidates allow mirroring production traffic to compare quality, latency, throughput, and cost before promoting.


### [Sidecars: A low-latency trust boundary for Sandboxes](https://yomu.fyi/post/sidecars-a-low-latency-trust-boundary-for-sandboxes.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Oct 1, 2026
- Type: Announcement

Modal introduces Sidecars, isolated containers that run alongside a main Sandbox on the same host, establishing a trust boundary between trusted and untrusted code. Sidecars enable communication across trust boundaries at least 3x faster than using separate Sandboxes, reducing latency for operation-heavy agent workloads. The feature addresses the security risk of running an agent’s harness and tool calls together, which creates the “lethal trifecta” of private data, untrusted content, and external communication. Existing solutions either move the harness to a remote service (adding network latency) or rely on egress controls that lack flexibility. Sidecars allow trusted services such as credential injection, proxy traffic, and harness logic to run locally alongside the Sandbox, with independent network policies and dynamic creation via the Modal SDK.


### [Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine](https://yomu.fyi/post/hitting-a-billion-tokens-per-minute-on-one-gpu-by-combining-a-query-pl.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 24, 2026
- Type: Problem & solution

Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.


### [How to serve trillions of tokens for trillion-parameter coding agents](https://yomu.fyi/post/how-to-serve-trillions-of-tokens-for-trillion-parameter-coding-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 23, 2026
- Type: Explainer

Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.


### [Product updates: Sandbox Sidecars, new models, a refreshed dashboard, and more](https://yomu.fyi/post/product-updates-sandbox-sidecars-new-models-a-refreshed-dashboard-and.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 14, 2026
- Type: Announcement

Modal added day-zero support for Kimi K3, Qwen 3.8, GLM 5.3, and GLM 5.3 Flash on Auto Endpoints, with custom DFlash speculators boosting throughput to 460 tokens per second. New environment-level budgets give Team and Enterprise workspaces per-environment compute limits, while the pricing multiplier for broad region selection dropped from 1.5x to 1.15x. The dashboard got a refreshed navigation, color system, and beta light mode, and Private Environments now support a default No Access role. Sandbox Sidecars are in public alpha, running additional containers via low-latency internal networking, and VM Sandboxes are in public beta with a real Linux kernel for Docker and eBPF. Sandbox directory snapshots and a new filesystem API are now GA, and regional Proxies allow selecting a region for static outbound IPs, while SDK updates improved log APIs and autoscaler reporting.


### [Modal is expanding in Europe with our new London office](https://yomu.fyi/post/modal-is-expanding-in-europe-with-our-new-london-office.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 2, 2026
- Type: Announcement

Modal is expanding in Europe with a new London office, its second in the region, representing a larger investment in the area. The company already has nearly 20 engineers in Stockholm who built core parts of Modal, including its custom file system and Sandboxes product. Hugh Killingbeck-Jones joins as VP EMEA to lead go-to-market teams in London, which the company considers the AI capital of Europe. European AI-native companies like Black Forest Labs and Legora, as well as global platforms like DoorDash, run on Modal for tasks such as model evaluation and agent testing. The company is hiring across sales, engineering, and other roles in London and Stockholm, and plans meetups across EMEA.


### [How Botika runs full-stack generative AI on Modal](https://yomu.fyi/post/how-botika-runs-full-stack-generative-ai-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 2, 2026
- Type: Problem & solution

Botika builds agentic e-commerce teams that automate visual production for global fashion brands, using proprietary foundation models and a 100-terabyte image dataset. The company runs its entire full-stack AI operation on Modal, with the data pipeline processing 100 terabytes at less than 1% error rate across thousands of containers. Before Modal, the team struggled with Kubernetes fleet management and slow GCP Batch jobs; now a single researcher can launch 50 experiments daily using Claude, and they set up reinforcement learning infrastructure in less than a day. Modal handles autoscaling, cold starts, and multi-node training with RDMA, allowing Botika to double traffic in seconds, and they serve about 15 models on L4s, L40S, A100s, and H100s. The infrastructure is not Modal-specific, enabling the team to run open-source and custom frameworks without modification.


### [Bringing serverless functions closer to the speed of wire](https://yomu.fyi/post/bringing-serverless-functions-closer-to-the-speed-of-wire.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Aug 4, 2026
- Type: Announcement

Modal’s legacy I/O plane routed all function inputs and outputs through us-east servers, adding up to ~200ms of network latency for calls from other regions. The new geographically distributed I/O plane, available in four or more regions, reduces end-to-end function call latencies by roughly 80ms at p50. The redesign involved moving non-critical work to background async processing, minimizing shared storage interactions on the hot path, and rewriting the server layer in Go for better concurrency. The team evaluated Valkey but chose Redis 7.1 because engine CPU usage spiked aggressively on Redis 7.2 onwards during input queue load testing. Users can further optimize by selecting the nearest routing\_region, keeping input and output payloads under 2MiB to avoid blob storage uploads, and batching small inputs to reduce network hops.


### [Kimi K3 by Moonshot now available on Modal](https://yomu.fyi/post/kimi-k3-by-moonshot-now-available-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 27, 2026
- Type: Announcement

Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads.


### [Devin Outposts on Modal](https://yomu.fyi/post/devin-outposts-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 21, 2026
- Type: Announcement

Devin, an AI software engineer by Cognition, plans, writes, tests, and ships code semi-autonomously. Cognition now extends Devin with Outposts, moving execution into user-controlled environments while reasoning stays in Cognition’s cloud. Modal is a launch partner with modal-devin, an open-source integration running Outpost sessions in Modal Sandboxes with pre-built custom images, fast cold-starts, snapshotting, and cost-efficient bursting. For example, a task that requires a GPU fails immediately in the default sandbox but succeeds on a Modal-backed Outpost that spins up a GPU on demand and scales to zero when done. One command, uvx modal-devin init, sets up secure infrastructure with a scheduler and worker functions, and each team can have its own outpost with separate credentials.


### [Scaling to 1 million concurrent sandboxes in seconds](https://yomu.fyi/post/scaling-to-1-million-concurrent-sandboxes-in-seconds.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 16, 2026
- Type: Problem & solution

Modal rebuilt its sandbox platform to support millions of concurrent sandboxes and tens of thousands of creations per second, removing all central bottlenecks from the control plane. The new architecture eliminates central coordination and durable datastores from the critical path, using horizontally scaled scheduling servers that run a fast load-balancing algorithm against in-memory cached worker state from Redis streams. Workers are contacted directly via RPC to create sandboxes, and sandbox metadata is written asynchronously, avoiding O(sandboxes) and O(nodes) operations that limited previous systems. In a benchmark, Modal created 1 million sandboxes in under a minute with median time-to-interactivity under half a second, though the long tail is attributed to kernel and network contention like the rtnl lock. The system is designed to scale without practical limits, supporting use cases like reinforcement learning and background agents that require massive concurrent sandbox creation.


### [Inkling by Thinking Machines now available on Modal](https://yomu.fyi/post/inkling-by-thinking-machines-now-available-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 15, 2026
- Type: Announcement

Thinking Machines released Inkling, a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. The 975B-parameter mixture-of-experts transformer activates 41B parameters per token, uses a 1M-token context window, and employs local attention: five sliding window layers for every global layer. Modal provided day-zero support with a custom DFlash block-diffusion speculator, achieving 250 tokens per second per user on 8x B200s with 67% faster interactivity than the model’s built-in multi-token prediction path. The DFlash drafter uses all local attention and causal layers, keeping cost flat as block size grows. Inkling is available today as an OpenAI-compatible shared endpoint through Modal Auto Endpoints with token-based pricing, allowing users to try the model interactively.


### [Modal is a computer](https://yomu.fyi/post/modal-is-a-computer.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 9, 2026
- Type: Explainer

Modal is a platform that runs programs of arithmetic and logic operations on information, making it a computer. It operates on hardware from dozens of cloud providers, building a virtualization layer that aggregates, isolates, and multiplexes demand for increased resource efficiency. Programs run in containers that receive inputs from the container runtime, similar to how a typical computer’s ALU fetches inputs from memory. Information in Modal lives in a tiered storage hierarchy, from container memory to object storage, with caches like page cache and ephemeral disk. Input and output pass through the Modal Input/Output Plane for functions and sandboxes, or the Routing Plane for HTTP servers, with direct Tunnels for extreme performance needs.


### [How to price serverless GPUs](https://yomu.fyi/post/how-to-price-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jul 6, 2026
- Type: Explainer

Modal’s engineering blog presents a cost model comparing serverless and reserved GPU pricing for AI workloads, centered on the peak-to-average demand ratio. Serverless GPUs charge only for usage, while reserved GPUs require long-term contracts for fixed capacity, often achieving under 30% utilization. The model shows that when the peak-to-average ratio exceeds the reservation discount rate (typically 2-5x), serverless GPUs become cheaper despite higher base rates, with real workloads like inference and training having ratios of 5-10x, making serverless an obvious cost win. An interactive tool allows teams to estimate costs using preset workloads or custom traffic patterns, defaulting to current B200 GPU rates on Modal and hyperscaler reservations. The post discusses model limitations, including assumptions of perfect prediction and instant allocation, and notes that serverless also reduces engineering costs through faster development and reduced dev/prod skew, while mixed strategies can combine baseline reservations with serverless bursting for high peak-to-average bursts.


### [Multi-token Residual Prediction](https://yomu.fyi/post/multi-token-residual-prediction.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 1, 2026
- Type: Explainer

Multi-Token Residual Prediction (MRP) adapts multi-token prediction for diffusion language models by training a small module to predict the residual between adjacent denoising steps rather than the full next-step distribution. This residual target is easier to learn, allowing a tiny 3-layer transformer to accurately predict corrections across multiple steps. In the static regime, MRP enables lossless speculative decoding with up to 1.56× throughput in SGLang, or direct decoding with a small quality cost. In the dynamic regime, MRP recovers up to +16 accuracy points lost by aggressive low-threshold decoding through a remasking mechanism. The module is trained on a frozen backbone and composes cleanly with existing inference methods.


[Older posts](https://yomu.fyi/company/modal/page/2.md)