Together AIAnnouncement
Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.
Swapping models in production forces a tradeoff between a hard swap that exposes all users to regressions at once and a DIY staged cutover that relies on manual traffic shifting and dashboard watching. Both approaches put a human in the loop as the safety mechanism, making rollbacks stressful and cold-starting the old model under pressure.
Together AI’s rollout system automates traffic migration between a source and target deployment on the same endpoint. Users choose a strategy: canary with configurable steps and metric gates, blue-green for a single-step cutover, or rolling for replica-by-replica swap. The platform handles scaling the target before traffic moves, runs health checks before each shift, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Metric gates (e.g., p95 latency regression) run after each canary step; if a gate trips, the rollout pauses automatically (SYSTEM_PAUSED) and waits for human approval to cancel or resume.
Rollouts move live traffic between source and target deployments in staged percentages, with health checks before every traffic shift, a propagation wait for routing caches, and source drain after traffic has moved.
Metric gates (e.g., p95 latency or error rate) run after each canary step; if a gate trips, the rollout enters SYSTEM_PAUSED and waits for human approval—it does not automatically fail or leave traffic in limbo.
A rolling strategy swaps replicas one by one, preserving total capacity without a traffic ramp, and is best suited for same-model engine or config changes rather than evaluating a new model on live traffic.
The rollout enters a SYSTEM_PAUSED state, which means the platform detected a failure (e.g., metric gate failed, capacity shortfall). It notifies the user and waits for human approval. There is no automatic failback; the user must decide to cancel or resume after investigating.
After traffic is shifted to the target, the system waits for a configured propagation interval before draining the source. This prevents stale global routing caches from sending requests to a source that has already shrunk, avoiding request failures.
Yes. You can pause a rollout explicitly with the --pause flag, putting it in a PAUSED state that holds at the current step. Resuming with --resume continues from the same step. The system can also auto-pause into SYSTEM_PAUSED when it detects a problem.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
