---
title: "Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents"
description: "VAKRA is an executable, tool-grounded benchmark for evaluating whether AI agents can complete enterprise-like workflows that combine API calls, document retrieval, and natural-language constraints. It provides more than 8,000 locally hosted APIs backed by real databases across 62 domains, plus domain-aligned documents, and uses full execution traces to assess multi-step behavior. The four capability groups cover API chaining, dashboard-API tool selection, multi-hop dashboard reasoning, and multi-hop multi-source reasoning with multi-turn dialogue and optional tool-use policies; tasks can require three to seven reasoning steps, while some segments specify one to twelve tool calls or up to five logical hops. Reported analyses find that models perform best on single-hop questions, degrade as API hops increase, and often violate policies or fail to retrieve enough information when constraints restrict the relevant source, exposing a gap between isolated tool competence and end-to-end reliability."
---

# Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

[Hugging Face](https://yomu.fyi/company/hugging-face) · Ankita Naik, danish, Ben, Anupama Murthi, Praveen Venkateswaran, Siyu, Ayhan Sebin · Apr 15, 2026

**Type:** Benchmark

## Summary

VAKRA is an executable, tool-grounded benchmark for evaluating whether AI agents can complete enterprise-like workflows that combine API calls, document retrieval, and natural-language constraints. It provides more than 8,000 locally hosted APIs backed by real databases across 62 domains, plus domain-aligned documents, and uses full execution traces to assess multi-step behavior. The four capability groups cover API chaining, dashboard-API tool selection, multi-hop dashboard reasoning, and multi-hop multi-source reasoning with multi-turn dialogue and optional tool-use policies; tasks can require three to seven reasoning steps, while some segments specify one to twelve tool calls or up to five logical hops. Reported analyses find that models perform best on single-hop questions, degrade as API hops increase, and often violate policies or fail to retrieve enough information when constraints restrict the relevant source, exposing a gap between isolated tool competence and end-to-end reliability.

## Context

Traditional benchmarks test isolated skills, while VAKRA evaluates compositional reasoning and reliable workflow completion across APIs, documents, execution traces, dialogue context, and tool-use constraints in enterprise-like environments.

## Approach / What changed

VAKRA uses an executable environment with more than 8,000 locally hosted APIs backed by real databases across 62 domains, domain-aligned document collections, and full execution traces. Its four capability groups test API chaining, dashboard-API tool selection, multi-hop API reasoning, and multi-source reasoning with multi-turn conversations and policy constraints.

## Takeaways

- VAKRA's API-chaining capability requires agents to initialize an instance with get\_data, then manipulate server-side data through tool calls; tasks can involve 1–12 calls.
- Capability 4 combines API calls and document retrieval at the hop level, with source decontamination ensuring that information needed for a hop is available in only one source.
- Models generally perform best on single-hop questions, while additional API hops, document retrieval, and policies restricting the relevant source reduce performance.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Hugging Face](https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis)
- Source URL: https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis
- Ingested by Yomu: 2026-08-29T01:47:44.160Z

[Read original post](https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis)