A New Framework for Evaluating Voice Agents (EVA)

Summary

EVA is an end-to-end framework for evaluating conversational voice agents across task accuracy and spoken conversational experience. It simulates multi-turn live-audio conversations with a user simulator, the evaluated voice agent, deterministic tool execution, validation, and metrics derived from recordings, transcripts, and tool-call logs. The framework reports EVA-A for Accuracy and EVA-X for Experience, combines code-based and LLM- or LALM-as-Judge metrics, and uses pass@3 and pass^3 across three trials to capture peak performance and consistency. Its initial release contains 50 reproducible English-language airline scenarios and 15 tools covering rebooking, itinerary changes, cancellations, standby, and vouchers. Results for 20 cascade and audio-native systems show a consistent tradeoff: agents stronger at task completion tend to provide worse experiences, while a substantial gap between pass@3 and pass^3 indicates inconsistent success.

Takeaways

1. 1

   EVA evaluates both task completion and conversational experience rather than treating them as separate concerns.

2. 2

   The initial dataset has 50 English-language airline scenarios and 15 tools spanning rebooking, itinerary changes, cancellations, same-day standby, and compensation vouchers.

3. 3

   Across 20 evaluated cascade and audio-native systems, higher task-completion performance generally corresponded to worse user experience, and pass@3 exceeded pass^3 substantially.