Scaling to 1 million concurrent sandboxes in seconds

Summary

Modal rebuilt its sandbox platform to support millions of concurrent sandboxes and tens of thousands of creations per second, removing all central bottlenecks from the control plane. The new architecture eliminates central coordination and durable datastores from the critical path, using horizontally scaled scheduling servers that run a fast load-balancing algorithm against in-memory cached worker state from Redis streams. Workers are contacted directly via RPC to create sandboxes, and sandbox metadata is written asynchronously, avoiding O(sandboxes) and O(nodes) operations that limited previous systems. In a benchmark, Modal created 1 million sandboxes in under a minute with median time-to-interactivity under half a second, though the long tail is attributed to kernel and network contention like the rtnl lock. The system is designed to scale without practical limits, supporting use cases like reinforcement learning and background agents that require massive concurrent sandbox creation.

Takeaways

1. 1

   The new architecture removes all central bottlenecks by eliminating durable datastores from the critical path of sandbox creation, using only in-memory state and asynchronous writes.

2. 2

   Scheduling is performed by a fleet of servers running a fast load-balancing algorithm against cached worker state, allowing horizontal scaling without coordination.

3. 3

   In a benchmark, Modal created 1 million sandboxes in under a minute with median time-to-interactivity under half a second, though the long tail is affected by kernel contention.

FAQ