Loading post…
Stack OverflowVarun JindalExplainer
Part 2: Evals as a deployment gate — and how to know when they drift
Summary
The post argues that evaluation for LLM systems must function as a deployment gate in CI, not as an ad-hoc notebook exercise, to catch regressions before they ship. It describes a structured golden set of test cases with explicit scoring, versioned alongside code, and auto-routed to slices based on request attributes, with per-slice pass-rate gates to prevent a 99% overall from hiding a 70% slice. The gate catches sharp quality drops (cliffs) but misses gradual degradation (slow leaks) across releases, which requires separate drift detection. Drift detection compares each run against a known-good baseline using a two-proportion z-test or difference-of-means test, flagging meaningful regressions even when the pass rate remains above the floor. Shadow evals sample a percentage of live decisions, score them out-of-band, and feed failures back into the golden set, while trend signals like human override rate and confidence shifts indicate emerging drift.
Context
This is the second level of a six-part series on running LLM systems in production, focusing on evaluation as a deployment gate and drift detection. It assumes the reader has basic familiarity with LLM systems and CI/CD practices.
Approach / What changed
Implement evaluation as a CI gate using a versioned golden set of test cases with explicit typed scoring, auto-routed to slices. Gate per slice on pass rate. Add drift detection by comparing each run against a known-good baseline using two-proportion z-tests or difference-of-means tests with minimum effect thresholds. Shadow-eval a sample of live traffic, score out-of-band, and feed failures into the golden set. Monitor trend signals like human override rate, confidence shifts, judge disagreement, and abstention rate.
Takeaways
Evals must be a CI gate that fails the build on regression, not a notebook exercise. Cases should be versioned data with explicit scoring, and the gate should run per slice to prevent a 99% overall from hiding a 70% slice.
Drift detection requires comparing each run against a known-good baseline using statistical significance (two-proportion z-test for rates, difference-of-means for continuous metrics) and a minimum effect threshold, not just a floor pass rate.
Shadow evals sample a deterministic hash of live decisions, score them out-of-band, and add failures to the golden set. Trend signals like human override rate, confidence distribution shifts, judge disagreement, and abstention rate indicate emerging drift.
FAQ
How do you prevent a global pass rate from hiding a bad slice?
Gate per slice, not just globally. Tag each case with its slice and auto-route cases to the matching slice. A 99% overall can hide a 70% slice, so enforce per-slice pass-rate thresholds.
What's the difference between a cliff and a slow leak in eval gates?
A cliff is a sharp quality drop caught by the pre-deploy gate. A slow leak is gradual degradation across releases, each step too small to trip the floor, so it requires drift detection that compares against a known-good baseline with statistical significance.
How do you sample live traffic for shadow evals without making tests flaky?
Use a deterministic hash of the decision ID, not random number generation, so sampling is reproducible. Set a sample rate (e.g., 5%) and use modulo on the hash to select decisions. This avoids flaky tests and allows consistent re-evaluation.
Source and freshness
- Source
- stackoverflow.blog
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.