---
title: "Part 2: Evals as a deployment gate — and how to know when they drift"
description: "The post argues that evaluation for LLM systems must function as a deployment gate in CI, not as an ad-hoc notebook exercise, to catch regressions before they ship. It describes a structured golden set of test cases with explicit scoring, versioned alongside code, and auto-routed to slices based on request attributes, with per-slice pass-rate gates to prevent a 99% overall from hiding a 70% slice. The gate catches sharp quality drops (cliffs) but misses gradual degradation (slow leaks) across releases, which requires separate drift detection. Drift detection compares each run against a known-good baseline using a two-proportion z-test or difference-of-means test, flagging meaningful regressions even when the pass rate remains above the floor. Shadow evals sample a percentage of live decisions, score them out-of-band, and feed failures back into the golden set, while trend signals like human override rate and confidence shifts indicate emerging drift."
---

# Part 2: Evals as a deployment gate — and how to know when they drift

[Stack Overflow](https://yomu.fyi/company/stack-overflow) · Varun Jindal · Oct 7, 2026

**Type:** Explainer

## Summary

The post argues that evaluation for LLM systems must function as a deployment gate in CI, not as an ad-hoc notebook exercise, to catch regressions before they ship. It describes a structured golden set of test cases with explicit scoring, versioned alongside code, and auto-routed to slices based on request attributes, with per-slice pass-rate gates to prevent a 99% overall from hiding a 70% slice. The gate catches sharp quality drops (cliffs) but misses gradual degradation (slow leaks) across releases, which requires separate drift detection. Drift detection compares each run against a known-good baseline using a two-proportion z-test or difference-of-means test, flagging meaningful regressions even when the pass rate remains above the floor. Shadow evals sample a percentage of live decisions, score them out-of-band, and feed failures back into the golden set, while trend signals like human override rate and confidence shifts indicate emerging drift.

## Context

This is the second level of a six-part series on running LLM systems in production, focusing on evaluation as a deployment gate and drift detection. It assumes the reader has basic familiarity with LLM systems and CI/CD practices.

## Approach / What changed

Implement evaluation as a CI gate using a versioned golden set of test cases with explicit typed scoring, auto-routed to slices. Gate per slice on pass rate. Add drift detection by comparing each run against a known-good baseline using two-proportion z-tests or difference-of-means tests with minimum effect thresholds. Shadow-eval a sample of live traffic, score out-of-band, and feed failures into the golden set. Monitor trend signals like human override rate, confidence shifts, judge disagreement, and abstention rate.

## Takeaways

- Evals must be a CI gate that fails the build on regression, not a notebook exercise. Cases should be versioned data with explicit scoring, and the gate should run per slice to prevent a 99% overall from hiding a 70% slice.
- Drift detection requires comparing each run against a known-good baseline using statistical significance (two-proportion z-test for rates, difference-of-means for continuous metrics) and a minimum effect threshold, not just a floor pass rate.
- Shadow evals sample a deterministic hash of live decisions, score them out-of-band, and add failures to the golden set. Trend signals like human override rate, confidence distribution shifts, judge disagreement, and abstention rate indicate emerging drift.

## FAQ

### How do you prevent a global pass rate from hiding a bad slice?

Gate per slice, not just globally. Tag each case with its slice and auto-route cases to the matching slice. A 99% overall can hide a 70% slice, so enforce per-slice pass-rate thresholds.

### What's the difference between a cliff and a slow leak in eval gates?

A cliff is a sharp quality drop caught by the pre-deploy gate. A slow leak is gradual degradation across releases, each step too small to trip the floor, so it requires drift detection that compares against a known-good baseline with statistical significance.

### How do you sample live traffic for shadow evals without making tests flaky?

Use a deterministic hash of the decision ID, not random number generation, so sampling is reproducible. Set a sample rate (e.g., 5%) and use modulo on the hash to select decisions. This avoids flaky tests and allows consistent re-evaluation.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Evaluation](https://yomu.fyi/topic/evaluation), [LLMs](https://yomu.fyi/topic/llm), [Testing](https://yomu.fyi/topic/testing)

- Source: [Stack Overflow](https://stackoverflow.blog/2026/10/07/evals-as-a-deployment-gate-and-how-to-know-when-they-drift)
- Source URL: https://stackoverflow.blog/2026/10/07/evals-as-a-deployment-gate-and-how-to-know-when-they-drift
- Ingested by Yomu: 2026-10-07T21:00:20.575Z

[Read original post](https://stackoverflow.blog/2026/10/07/evals-as-a-deployment-gate-and-how-to-know-when-they-drift)