Bringing serverless functions closer to the speed of wire

Summary

Modal’s legacy I/O plane routed all function inputs and outputs through us-east servers, adding up to ~200ms of network latency for calls from other regions. The new geographically distributed I/O plane, available in four or more regions, reduces end-to-end function call latencies by roughly 80ms at p50. The redesign involved moving non-critical work to background async processing, minimizing shared storage interactions on the hot path, and rewriting the server layer in Go for better concurrency. The team evaluated Valkey but chose Redis 7.1 because engine CPU usage spiked aggressively on Redis 7.2 onwards during input queue load testing. Users can further optimize by selecting the nearest routing_region, keeping input and output payloads under 2MiB to avoid blob storage uploads, and batching small inputs to reduce network hops.

Takeaways

1. 1

   The new regional I/O plane reduces p50 function call latency by roughly 80ms compared to the legacy us-east-only routing.

2. 2

   Rewriting the server layer in Go improved concurrency for gRPC handling over async Python, which suffered from event loop lag under CPU-intensive operations.

3. 3

   Keep input and output payloads under 2MiB to avoid extra blob storage uploads; batch small inputs into one larger input to reduce network hops.

FAQ