Prevent, Verify, Contain: The Three Bills of AI Safety
AI safety is a budget: prevention, verification, and containment each buy different protection at a different cost. This map shows what each control costs and when it pays off.
Loading…
Topic
Evaluation
10 posts about Evaluation. Every summary links to the original.
- 10 posts
- 7 companies
- 5 explainers
See the idea
Evaluation explainers
How to Run an Agent Loop That Actually Converges
A four-checkpoint how-to for agent loops: name a measurable job, bound the tool surface, refresh sandbox fixtures from real traffic, and verify at agent speed.
Agentic Development: The Four Parts of a Delegation Loop
Agentic development turns AI assistance into delegated work: a harness runs the agents, skills supply domain knowledge, a sandbox bounds execution, and evals verify the result.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Apertus 1.5 Release Notes
Apertus 1.5 extends the 8B and 70B open-weight models with image and audio understanding, a thinking mode, and a 262,144-token context window, released under Apache 2.0. The models are a continued pretraining of Apertus 1.0, adding 4 trillion tokens to the 8B model and 2 trillion to the 70B model, for totals of 19T and 17T respectively. Multimodality uses an early-fusion tokenizer with Emu3.5 for images and WavTokenizer for audio, keeping the architecture unchanged and training at text-only speed. Post-training includes SFT, long-context SFT, reinforcement learning with verifiable rewards, DPO, and charter alignment via SDPO, with all data and recipes publicly documented. Evaluation results show competitive performance on reasoning, coding, and multilingual tasks, with the 8B model matching GPT-OSS-20B on translation and outperforming Claude Haiku 4.5, reported transparently via the EvalEval Coalition’s EEE schema.
Evaluating AI agents: moving past the vibe check
Manual vibe checks for AI agents miss silent failures like retry loops, token waste, and hallucinated parameters while plausible final answers mask them. The post outlines a structured evaluation pipeline using the open-source agent-eval toolkit, starting with critical user journeys (CUJs) that define success in concrete, testable clauses. Synthetic user simulations (via ADK) generate multi-turn scenarios for initial trace collection, which developers grade with a ‘red pen’ before automating checks. Four layers of verification are recommended: unit tests, deterministic metrics, LLM-as-a-judge with boolean rubrics (avoiding unreliable 1-to-5 scales), and integration tests, all standardized on OpenInference/OpenTelemetry traces. The rollout progresses from Day 0 manual review through Day 1 automated checks to Day 2 CI/CD gates and shared dashboards, with real production traces fed back into the golden dataset.
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.
Production-grade LLMs and agents: a field guide
Production-grade LLM agents require deterministic machinery around the probabilistic model, not a larger model. A maturity model defines six levels: determinism, evaluation, confidence, safety and governance, operability, and production-grade build, each assuming the one below. Teams should fix their weakest level in order, starting with determinism before evals, and contain the model to a single node in a fixed graph where it proposes actions a separate component applies after approval. A self-assessment checklist with items like evals as a CI gate and layered guardrails helps teams identify their current level, with fewer than half ticked indicating they are earlier than the demo suggests. The post links to a series of 25 deep-dive articles covering each rung of the maturity model.
The Outer Loop, Insights First: An Ambient Quality Agent That Diagnoses Your Production Agent
AI agents in production often pass health checks while silently failing at real tasks, such as confirming a seat without checking availability. Google and ADK released AQuA (Ambient Quality Agent), a sidecar system that continuously sweeps production agent trajectories from Cloud Trace, Cloud Logging, or BigQuery outside the request path. AQuA samples up to 1,000 recent sessions, grades each against a developer-defined goal and a nine-point breakdown checklist using a single-pass judge (or optional Gemini AutoRaters), clusters failures by mechanism, verifies clusters with a second model, and tracks them in BigQuery. Developers can trigger root-cause analysis that reads failing trajectories alongside the immutable deploy-time source snapshot, citing exact code lines without auto-applying edits. The system runs unattended, sweeps after each deploy or on demand, and produces verified failure clusters to feed back into the inner development loop, solving quality degradation that standard health checks miss.
Part 2: Evals as a deployment gate — and how to know when they drift
The post argues that evaluation for LLM systems must function as a deployment gate in CI, not as an ad-hoc notebook exercise, to catch regressions before they ship. It describes a structured golden set of test cases with explicit scoring, versioned alongside code, and auto-routed to slices based on request attributes, with per-slice pass-rate gates to prevent a 99% overall from hiding a 70% slice. The gate catches sharp quality drops (cliffs) but misses gradual degradation (slow leaks) across releases, which requires separate drift detection. Drift detection compares each run against a known-good baseline using a two-proportion z-test or difference-of-means test, flagging meaningful regressions even when the pass rate remains above the floor. Shadow evals sample a percentage of live decisions, score them out-of-band, and feed failures back into the golden set, while trend signals like human override rate and confidence shifts indicate emerging drift.
The Agent Said It Was Done. The Database Disagreed.
ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data.
Master your skills: Building skills you can trust
Skills in AI agent projects silently misfire when their trigger descriptions are passive or too general, causing the agent to skip using them. The authoring tool skill-creator, from Anthropic’s skills collection, provides an eval loop that generates positive and negative test prompts and iterates on description accuracy using a training and held-out test set to avoid overfitting. These tests are ephemeral, so the open-source framework promptfoo extends them into permanent configuration files that can be committed alongside skills. Promptfoo’s agent SDK providers test through the actual agent runtime and include a skill-used assertion to confirm proper routing, and its YAML-based configuration fits naturally into version control. The final step is wiring promptfoo evals into CI, using GitHub Actions with path filters to run on pull requests that modify skills, catching regressions before they ship and costing roughly $0.10-0.15 per eval.
ShopGym: Realistic, reproducible sandboxes for shopping agents (2026) - Shopify
ShopGym converts live storefronts into self-contained, resettable sandbox shops for training and evaluating shopping agents, addressing challenges from changing products and bot detection. Its ShopArena layer separates exploration from generation using anonymized specifications and coding agents with execution-verification loops that repair failures without regenerating the whole shop. ShopGuru then synthesizes benchmark tasks across seven skill categories using rule-based and LLM generators grounded in each shop’s catalog, navigation, and policies. Validation over 224 tasks across six shops, three synthetic and three real, showed synthetic shops retain key structural properties and agent performance positively correlates with live storefronts when tested with BrowserGym and an internal system. The framework aims to provide reproducible environments for agents to fail and learn before interacting with live stores, with plans to expand to more retail domains and task types.
How to Evaluate AI Agents From Tool Calls to Task Completion
Standard LLM benchmarks are insufficient for evaluating AI agents because they score static outputs rather than multi-step tool-call sequences against a live environment. The post explains that agent evaluation requires two scoring layers: step-level (process scoring) checks each call’s validity and usefulness, while end-to-end (E2E) scoring checks only the final environment state. A trace logs every step and environment state, and metrics roll up through a hierarchy of benchmark, trial, task, turn, and step. Key metrics include task success rate, consistency, tool-call precision, argument accuracy, steps per success, and cost per success. The post argues that benchmarks are converging on tool use because deployed agents always use tools, and enterprise benchmarks should measure task completion against real APIs and policies.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.