---
title: "Proxy"
description: "2 posts about Proxy, summarised, each linking to the original."
---

# Proxy
> 2 posts about Proxy, summarised, each linking to the original.

## Articles

### [Endpoint Candidates: evaluate and promote inference endpoints](https://yomu.fyi/post/endpoint-candidates-evaluate-and-promote-inference-endpoints.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Amit Lohe, Deven Navani, Garrett O'Brien
- Published: Oct 12, 2026
- Type: Announcement

Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.


### [Modal's serverless Servers](https://yomu.fyi/post/modal-s-serverless-servers.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 25, 2026
- Type: Problem & solution

Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.