Loading post…
Together AITutorial
A/B test models in production
Summary
A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.
Context
The guide presents endpoint-level A/B experiments for comparing candidate models with real users. It contrasts this approach with application-layer feature flags, hardcoded model routing, and shadow traffic, then covers routing, ramping, measurement, promotion, deletion, and operational edge cases.
Approach / What changed
Attach an experiment to an endpoint with exactly one control and one or more variants. Keep only the control in the endpoint’s base traffic split, with variants created and READY at zero weight; assign experiment percentages that sum to 100%. Update the full member list for each ramp, protect updates with an etag, join deployment-level metrics to product outcomes through response metadata, and use a blue-green rollout before deleting a winning experiment.
Takeaways
Experiment percentages are fixed traffic shares independent of replica counts, while base traffic-split weights are per-ready-replica and can follow capacity.
Variants must carry zero weight in the endpoint’s base split; otherwise they can receive additional capacity-weighted traffic and distort measurements.
Deleting an experiment sends traffic back to the base split. A winning variant should be promoted first, while a losing test can be ended by deleting the experiment.
FAQ
Why must variant deployments have zero weight in the endpoint’s base split?
The experiment should own all traffic sent to variants. If a variant also receives capacity-weighted traffic from the base split, the observed experiment percentages and measurements become inaccurate.
How does the experiment connect model exposure to product quality?
Each response includes the deployment ID that served it. Logging that ID with ratings, retries, task completions, or other product signals lets analytics join user outcomes to platform metrics such as latency and errors.
What happens when an experiment is deleted?
Experiment routing disappears and traffic follows the endpoint’s base traffic split. If the control remains there, traffic returns entirely to it; after a rollout, the base split can point to the promoted winner.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.