ModalAmit Lohe, Deven Navani, Garrett O'BrienAnnouncement
Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.
Latency, throughput, and cost optimization are central to inference engineering; Modal provides Endpoint Candidates to enable systematic evaluation of these metrics.
The system extends Modal’s Pingora-based proxy to fork each request stream into separate bounded buffers for every Candidate receiving a copy. Mirrors read independently; if a buffer fills or the shared per-process chunk budget is exceeded, the proxy cancels that mirrored request while production and other mirrors continue. Mirrring is best-effort by design.
Endpoint Candidates allow staging alternative deployments that mirror a sample of production traffic to evaluate quality, latency, throughput, and cost before promotion.
The Pingora-based proxy forks request streams into bounded buffers per Candidate and shares body chunks across mirrors with a per-process budget; mirroring is best-effort and canceled if buffer fills or budget exceeded.
Candidates can be promoted from the dashboard, and other Candidates remain available for independent testing; the feature integrates with Auto Endpoints for automated optimization validation.
Start from an existing Dedicated Endpoint, select Create candidate, then choose an open model from Modal's catalog or import custom weights from Hugging Face or a Modal Volume. Configure the percentage of production traffic to mirror.
Each Candidate has a separate bounded buffer, and chunks are shared across mirrors within a per-process memory budet. If a buffer fills or the budet is exceeded, the proxy cancels that mirrored request while production continues.
Endpoint Candidates use standard Modal compute pricing, including compute used for mirrored requests. There is no additional charge for the feature itself.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Amazon Bedrock is the managed reasoning tier in an agent architecture, not the whole system. This layered view shows the pipelines, governed tables, query layer, and guardrails around it.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
