---
title: "Modal's serverless Servers"
description: "Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning."
---

# Modal's serverless Servers

[Modal](https://yomu.fyi/company/modal) · Jun 25, 2026

**Type:** Problem & solution

## Summary

Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.

## Context

Modal needed an HTTP serving solution with minimum overhead for ultra-low-latency applications like LLM inference, as their existing Web Functions added 39ms p50 latency due to built-in queueing and platform-managed request lifecycle.

## Approach / What changed

Built a new HTTP serving layer (Servers) using a streaming edge proxy (Envoy), a stateless in-house proxy (fprs) for routing and load balancing, and a compute load balancer. The system avoids all network calls on the request path and materializes minimal state in the routing layer, pushing queueing and retries to the application layer.

## Takeaways

- Modal Servers achieve 6ms p50 latency by removing all control-plane lookups and queueing from the hot path, pushing those responsibilities to the application layer.
- The architecture follows two principles: maximize resource sharing while minimizing interference, and make no network calls on the request path.
- Servers support proxy auth to block unauthorized requests at the routing layer, preventing costly container spin-ups for denial-of-service attacks.

## FAQ

### How do Modal Servers differ from Modal Web Functions?

Web Functions include built-in queueing and a platform-managed request lifecycle for robustness, adding ~39ms p50 latency. Servers remove that overhead, reducing p50 latency to ~6ms, but require applications to handle queueing and load-shedding (e.g., via 503 responses).

### What happens if no Server replica is available?

Clients receive a 503 Service Unavailable response, the same as if the service didn't exist. Queueing and retries are not handled at the routing layer; they are pushed to the container application.

### What is the routing URL format for Modal Servers?

The URL follows the pattern https://\[domain\].\[routing\_region\].modal.direct/\[path\], where \[domain\] identifies the Server (autoscaling pool), \[routing\_region\] selects a regional proxy deployment, and \[path\] identifies the resource.

**Tags:** [Envoy](https://yomu.fyi/topic/envoy), [Latency](https://yomu.fyi/topic/latency), [LLMs](https://yomu.fyi/topic/llm), [Proxy](https://yomu.fyi/topic/proxy), [Serverless](https://yomu.fyi/topic/serverless)

- Source: [Modal](https://modal.com/blog/serverless-servers)
- Source URL: https://modal.com/blog/serverless-servers
- Ingested by Yomu: 2026-09-20T07:47:05.414Z

[Read original post](https://modal.com/blog/serverless-servers)