---
title: "Evaluating AI agents: moving past the vibe check"
description: "Manual vibe checks for AI agents miss silent failures like retry loops, token waste, and hallucinated parameters while plausible final answers mask them. The post outlines a structured evaluation pipeline using the open-source agent-eval toolkit, starting with critical user journeys (CUJs) that define success in concrete, testable clauses. Synthetic user simulations (via ADK) generate multi-turn scenarios for initial trace collection, which developers grade with a ‘red pen’ before automating checks. Four layers of verification are recommended: unit tests, deterministic metrics, LLM-as-a-judge with boolean rubrics (avoiding unreliable 1-to-5 scales), and integration tests, all standardized on OpenInference/OpenTelemetry traces. The rollout progresses from Day 0 manual review through Day 1 automated checks to Day 2 CI/CD gates and shared dashboards, with real production traces fed back into the golden dataset."
---

# Evaluating AI agents: moving past the vibe check

[Google](https://yomu.fyi/company/google) · Hugo Selbie, Dani Zamora, Jesse Carah · Oct 15, 2026

**Type:** Problem & solution

## Summary

Manual vibe checks for AI agents miss silent failures like retry loops, token waste, and hallucinated parameters while plausible final answers mask them. The post outlines a structured evaluation pipeline using the open-source agent-eval toolkit, starting with critical user journeys (CUJs) that define success in concrete, testable clauses. Synthetic user simulations (via ADK) generate multi-turn scenarios for initial trace collection, which developers grade with a ‘red pen’ before automating checks. Four layers of verification are recommended: unit tests, deterministic metrics, LLM-as-a-judge with boolean rubrics (avoiding unreliable 1-to-5 scales), and integration tests, all standardized on OpenInference/OpenTelemetry traces. The rollout progresses from Day 0 manual review through Day 1 automated checks to Day 2 CI/CD gates and shared dashboards, with real production traces fed back into the golden dataset.

## Context

Teams often rely on ad-hoc manual checks (vibe checks) for AI agents, which miss silent failures like retry loops, tool-call hallucinations, and excessive token consumption because the final output appears plausible.

## Approach / What changed

Define critical user journeys (CUJs) with testable clauses, generate synthetic user simulations via ADK, apply four layers of checks (unit tests, deterministic metrics, LLM-as-a-judge with boolean rubrics, integration tests), standardize traces using OpenInference/OpenTelemetry, and progressively roll out from Day 0 manual review through Day 1 automated checks to Day 2 CI/CD gates with shared dashboards and production trace feedback.

## Takeaways

- AI agent failures are often invisible in final text output: retry loops, hallucinated tool parameters, and excessive token usage can occur while the agent returns a plausible answer. Evaluation must inspect intermediate traces (tool calls, reasoning steps) and operational metrics like latency and cost.
- LLM judges are unreliable on qualitative 1-to-5 scales because the boundary between scores shifts between runs. Convert rubrics to boolean pass/fail questions, such as ‘Did the agent verify the account ID before calling the refund tool?’, paired with deterministic checks for tool parameters and reference comparisons.
- To reuse evaluation across different agent frameworks (LangGraph, Google ADK, custom), decouple the pipeline from the orchestration layer by standardizing OpenTelemetry traces on OpenInference semantic conventions. The agent-eval toolkit supports this natively and routes traces to the Agent Evaluation service on the Gemini platform.

## FAQ

### What are the issues with manual vibe checks for AI agents?

Running three or four prompts in a terminal only shows the final text response. An agent can loop on retries, hallucinate tool parameters, burn token budgets, or trigger duplicate side effects while returning a polite answer. Automated trace evaluation checks intermediate tool calls, state transitions, latency, and cost.

### How do I evaluate an AI agent from scratch?

Treat evaluation as a progressive rollout: Day 0 break CUJs into test scenarios, run ADK simulations, and read 5-20 traces with a 'red pen'. Day 1 turn notes into automated checks. Day 2 lock checks into CI/CD gates. Replace synthetic data with real production traces over time.

### Why use boolean pass/fail checks instead of 1-to-5 scales?

LLM judges are inconsistent on 1-to-5 scales because the boundary between '3' and '4' shifts between runs. With Adaptive Rubrics, split goals into strict Yes/No questions like 'Did the agent verify the account ID before refunding?' Pair with deterministic metrics for tool parameters and trace assertions for latency and token budgets.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Evaluation](https://yomu.fyi/topic/evaluation), [LangGraph](https://yomu.fyi/topic/langgraph), [OpenTelemetry](https://yomu.fyi/topic/opentelemetry)

- Source: [Google](https://developers.googleblog.com/evaluating-ai-agents-moving-past-the-vibe-check)
- Source URL: https://developers.googleblog.com/evaluating-ai-agents-moving-past-the-vibe-check
- Ingested by Yomu: 2026-10-15T15:00:19.107Z

[Read original post](https://developers.googleblog.com/evaluating-ai-agents-moving-past-the-vibe-check)