---
title: "A New Framework for Evaluating Voice Agents (EVA)"
description: "EVA is an end-to-end framework for evaluating conversational voice agents across task accuracy and spoken conversational experience. It simulates multi-turn live-audio conversations with a user simulator, the evaluated voice agent, deterministic tool execution, validation, and metrics derived from recordings, transcripts, and tool-call logs. The framework reports EVA-A for Accuracy and EVA-X for Experience, combines code-based and LLM- or LALM-as-Judge metrics, and uses pass@3 and pass^3 across three trials to capture peak performance and consistency. Its initial release contains 50 reproducible English-language airline scenarios and 15 tools covering rebooking, itinerary changes, cancellations, standby, and vouchers. Results for 20 cascade and audio-native systems show a consistent tradeoff: agents stronger at task completion tend to provide worse experiences, while a substantial gap between pass@3 and pass^3 indicates inconsistent success."
---

# A New Framework for Evaluating Voice Agents (EVA)

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tara Bogavelli, Gabrielle Gauthier Melancon, Katrina Stankiewicz, Nifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, Hari Subramani, Fanny Riols · Mar 24, 2026

**Type:** Benchmark

## Summary

EVA is an end-to-end framework for evaluating conversational voice agents across task accuracy and spoken conversational experience. It simulates multi-turn live-audio conversations with a user simulator, the evaluated voice agent, deterministic tool execution, validation, and metrics derived from recordings, transcripts, and tool-call logs. The framework reports EVA-A for Accuracy and EVA-X for Experience, combines code-based and LLM- or LALM-as-Judge metrics, and uses pass@3 and pass^3 across three trials to capture peak performance and consistency. Its initial release contains 50 reproducible English-language airline scenarios and 15 tools covering rebooking, itinerary changes, cancellations, standby, and vouchers. Results for 20 cascade and audio-native systems show a consistent tradeoff: agents stronger at task completion tend to provide worse experiences, while a substantial gap between pass@3 and pass^3 indicates inconsistent success.

## Context

Existing evaluation efforts assess speech understanding, perceived speech quality, conversational dynamics, or agentic capabilities separately. They do not jointly evaluate whether a voice agent completes a task correctly and communicates accurately, concisely, and naturally throughout a complete, multi-step spoken workflow.

## Approach / What changed

EVA uses a bot-to-bot audio architecture for multi-turn conversations. A goal-driven user simulator interacts with a voice agent, which can use deterministic tools backed by a scenario database. Validators filter out invalid simulator executions, while code-based metrics and LLM- or LALM-as-Judge metrics assess conversation recordings, transcripts, and tool-call logs using EVA-A, EVA-X, and diagnostic measures.

## Takeaways

- EVA evaluates both task completion and conversational experience rather than treating them as separate concerns.
- The initial dataset has 50 English-language airline scenarios and 15 tools spanning rebooking, itinerary changes, cancellations, same-day standby, and compensation vouchers.
- Across 20 evaluated cascade and audio-native systems, higher task-completion performance generally corresponded to worse user experience, and pass@3 exceeded pass^3 substantially.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Hugging Face](https://huggingface.co/blog/ServiceNow-AI/eva)
- Source URL: https://huggingface.co/blog/ServiceNow-AI/eva
- Ingested by Yomu: 2026-08-29T01:46:47.829Z

[Read original post](https://huggingface.co/blog/ServiceNow-AI/eva)