Loading post…
ModalProblem & solution
Modal's serverless Servers
Summary
Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.
Context
Modal needed an HTTP serving solution with minimum overhead for ultra-low-latency applications like LLM inference, as their existing Web Functions added 39ms p50 latency due to built-in queueing and platform-managed request lifecycle.
Approach / What changed
Built a new HTTP serving layer (Servers) using a streaming edge proxy (Envoy), a stateless in-house proxy (fprs) for routing and load balancing, and a compute load balancer. The system avoids all network calls on the request path and materializes minimal state in the routing layer, pushing queueing and retries to the application layer.
Takeaways
Modal Servers achieve 6ms p50 latency by removing all control-plane lookups and queueing from the hot path, pushing those responsibilities to the application layer.
The architecture follows two principles: maximize resource sharing while minimizing interference, and make no network calls on the request path.
Servers support proxy auth to block unauthorized requests at the routing layer, preventing costly container spin-ups for denial-of-service attacks.
FAQ
How do Modal Servers differ from Modal Web Functions?
Web Functions include built-in queueing and a platform-managed request lifecycle for robustness, adding ~39ms p50 latency. Servers remove that overhead, reducing p50 latency to ~6ms, but require applications to handle queueing and load-shedding (e.g., via 503 responses).
What happens if no Server replica is available?
Clients receive a 503 Service Unavailable response, the same as if the service didn't exist. Queueing and retries are not handled at the routing layer; they are pushed to the container application.
What is the routing URL format for Modal Servers?
The URL follows the pattern https://[domain].[routing_region].modal.direct/[path], where [domain] identifies the Server (autoscaling pool), [routing_region] selects a regional proxy deployment, and [path] identifies the resource.
Source and freshness
- Source
- modal.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.