Part 3: Knowing when your agent doesn’t know: the confidence layer

Summary

A production LLM system must know when it is uncertain, not just produce answers. The confidence layer composes a score from four independent signals — model self-score, deterministic verification, judge agreement, and historical accuracy — with the model’s own claim deliberately weighted lowest. That raw score is calibrated via isotonic regression against empirical accuracy so that a score of 0.9 means right roughly 90% of the time. A per-slice threshold decides automation versus human review, and an abstain floor escalates inputs too uncertain to even draft. An independent judge evaluates high-stakes decisions, feeding its agreement into the confidence score and flagging disagreements. Human handoffs show the draft, reasoning, evidence, and why it was routed, and every human action is captured as signal to tune thresholds and improve the agent.

Takeaways

1. 1

   Compose a confidence score from multiple independent signals (model self-score, verification, judge agreement, historical accuracy) with model_self weighted lowest to prevent confidently wrong answers from clearing the automation bar.

2. 2

   Calibrate the raw composed score via isotonic regression against empirical accuracy on a rolling window; a score of 0.9 must mean right ~90% of the time, and calibration must be rechecked regularly as it drifts.

3. 3

   Route decisions per slice with a threshold set from calibration data, and make abstention a first-class outcome below a floor — weird inputs automatically escalate to a human rather than producing a bad draft.

FAQ