# How to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations

[Salesforce](https://yomu.fyi/company/salesforce) · Scott Nyberg · Aug 17, 2026

**Type:** Problem & solution

## Summary

Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.

## Context

Traditional LLM evaluation relies on generated conversational output, which fails for production AI agents that use tool calling. Models can generate text claiming an action succeeded even when tool calls were omitted, execution sequences were violated, or external systems were left unchanged or corrupted.

## Approach / What changed

CRMAgentBench evaluates multi-turn workflows in shared, stateful environments using all-or-nothing assertions that check tool selection, argument accuracy, execution sequence, final system state, and the absence of collateral damage or forbidden tools. System reliability across multiple attempts is measured using pass^k metrics.

## Takeaways

- Conversational evaluations can register false successes because LLMs may narrate that a task was completed without actually executing the necessary tool calls.
- Strict all-or-nothing evaluation requires verifying correct tool selection, argument precision, call sequence, final system state, and the absence of forbidden actions or collateral changes.
- Pass^k metrics evaluate whether an agent consistently succeeds across k independent runs, distinguishing dependable models from high-variance models that pass@k metrics reward.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://engineering.salesforce.com/how-to-evaluate-production-ai-agents-measure-system-outcomes-not-conversations)