---
title: "A/B test models in production"
description: "A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test."
---

# A/B test models in production

[Together AI](https://yomu.fyi/company/together-ai) · Aug 17, 2026

**Type:** Tutorial

## Summary

A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.

## Context

The guide presents endpoint-level A/B experiments for comparing candidate models with real users. It contrasts this approach with application-layer feature flags, hardcoded model routing, and shadow traffic, then covers routing, ramping, measurement, promotion, deletion, and operational edge cases.

## Approach / What changed

Attach an experiment to an endpoint with exactly one control and one or more variants. Keep only the control in the endpoint’s base traffic split, with variants created and READY at zero weight; assign experiment percentages that sum to 100%. Update the full member list for each ramp, protect updates with an etag, join deployment-level metrics to product outcomes through response metadata, and use a blue-green rollout before deleting a winning experiment.

## Takeaways

- Experiment percentages are fixed traffic shares independent of replica counts, while base traffic-split weights are per-ready-replica and can follow capacity.
- Variants must carry zero weight in the endpoint’s base split; otherwise they can receive additional capacity-weighted traffic and distort measurements.
- Deleting an experiment sends traffic back to the base split. A winning variant should be promoted first, while a losing test can be ended by deleting the experiment.

## FAQ

### Why must variant deployments have zero weight in the endpoint’s base split?

The experiment should own all traffic sent to variants. If a variant also receives capacity-weighted traffic from the base split, the observed experiment percentages and measurements become inaccurate.

### How does the experiment connect model exposure to product quality?

Each response includes the deployment ID that served it. Logging that ID with ratings, retries, task completions, or other product signals lets analytics join user outcomes to platform metrics such as latency and errors.

### What happens when an experiment is deleted?

Experiment routing disappears and traffic follows the endpoint’s base traffic split. If the control remains there, traffic returns entirely to it; after a rollout, the base split can point to the promoted winner.

**Tags:** [Experimentation](https://yomu.fyi/topic/experimentation), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [Model Serving](https://yomu.fyi/topic/model-serving)

- Source: [Together AI](https://www.together.ai/blog/a-b-test-models-in-production)
- Source URL: https://www.together.ai/blog/a-b-test-models-in-production
- Ingested by Yomu: 2026-09-19T10:13:45.492Z

[Read original post](https://www.together.ai/blog/a-b-test-models-in-production)