Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

Summary

VAKRA is an executable, tool-grounded benchmark for evaluating whether AI agents can complete enterprise-like workflows that combine API calls, document retrieval, and natural-language constraints. It provides more than 8,000 locally hosted APIs backed by real databases across 62 domains, plus domain-aligned documents, and uses full execution traces to assess multi-step behavior. The four capability groups cover API chaining, dashboard-API tool selection, multi-hop dashboard reasoning, and multi-hop multi-source reasoning with multi-turn dialogue and optional tool-use policies; tasks can require three to seven reasoning steps, while some segments specify one to twelve tool calls or up to five logical hops. Reported analyses find that models perform best on single-hop questions, degrade as API hops increase, and often violate policies or fail to retrieve enough information when constraints restrict the relevant source, exposing a gap between isolated tool competence and end-to-end reliability.

Takeaways

1. 1

   VAKRA's API-chaining capability requires agents to initialize an instance with get_data, then manipulate server-side data through tool calls; tasks can involve 1–12 calls.

2. 2

   Capability 4 combines API calls and document retrieval at the hop level, with source decontamination ensuring that information needed for a hop is available in only one source.

3. 3

   Models generally perform best on single-hop questions, while additional API hops, document retrieval, and policies restricting the relevant source reduce performance.