---
title: "Canary rollouts: upgrade models in production without downtime"
description: "Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests."
---

# Canary rollouts: upgrade models in production without downtime

[Together AI](https://yomu.fyi/company/together-ai) · Sep 22, 2026

**Type:** Announcement

## Summary

Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.

## Context

Swapping models in production forces a tradeoff between a hard swap that exposes all users to regressions at once and a DIY staged cutover that relies on manual traffic shifting and dashboard watching. Both approaches put a human in the loop as the safety mechanism, making rollbacks stressful and cold-starting the old model under pressure.

## Approach / What changed

Together AI’s rollout system automates traffic migration between a source and target deployment on the same endpoint. Users choose a strategy: canary with configurable steps and metric gates, blue-green for a single-step cutover, or rolling for replica-by-replica swap. The platform handles scaling the target before traffic moves, runs health checks before each shift, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Metric gates (e.g., p95 latency regression) run after each canary step; if a gate trips, the rollout pauses automatically (SYSTEM\_PAUSED) and waits for human approval to cancel or resume.

## Takeaways

- Rollouts move live traffic between source and target deployments in staged percentages, with health checks before every traffic shift, a propagation wait for routing caches, and source drain after traffic has moved.
- Metric gates (e.g., p95 latency or error rate) run after each canary step; if a gate trips, the rollout enters SYSTEM\_PAUSED and waits for human approval—it does not automatically fail or leave traffic in limbo.
- A rolling strategy swaps replicas one by one, preserving total capacity without a traffic ramp, and is best suited for same-model engine or config changes rather than evaluating a new model on live traffic.

## FAQ

### What happens when a metric gate fails?

The rollout enters a SYSTEM\_PAUSED state, which means the platform detected a failure (e.g., metric gate failed, capacity shortfall). It notifies the user and waits for human approval. There is no automatic failback; the user must decide to cancel or resume after investigating.

### How does propagation wait prevent errors?

After traffic is shifted to the target, the system waits for a configured propagation interval before draining the source. This prevents stale global routing caches from sending requests to a source that has already shrunk, avoiding request failures.

### Can I pause and resume a running rollout?

Yes. You can pause a rollout explicitly with the --pause flag, putting it in a PAUSED state that holds at the current step. Resuming with --resume continues from the same step. The system can also auto-pause into SYSTEM\_PAUSED when it detects a problem.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [MLOps](https://yomu.fyi/topic/mlops), [Model Serving](https://yomu.fyi/topic/model-serving), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Together AI](https://www.together.ai/blog/canary-rollouts-upgrade-models-in-production-without-downtime)
- Source URL: https://www.together.ai/blog/canary-rollouts-upgrade-models-in-production-without-downtime
- Ingested by Yomu: 2026-09-23T00:00:14.507Z

[Read original post](https://www.together.ai/blog/canary-rollouts-upgrade-models-in-production-without-downtime)