---
title: "The Agent Said It Was Done. The Database Disagreed."
description: "ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data."
---

# The Agent Said It Was Done. The Database Disagreed.

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tuhin Kundu · Oct 3, 2026

**Type:** Benchmark

## Summary

ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data.

## Context

ThinkingBox is a benchmark from Microsoft and Hugging Face that evaluates AI agents on the database state and side effects they leave behind, rather than on tool calls or final responses. It runs each of 507 stateful business workflows 20 times from identical clean backends and reports pass@1, pass@20, and observed 20/20 consistency. The benchmark reveals a significant gap between single-attempt capability and consistent reliability across models.

## Approach / What changed

ThinkingBox runs agents against isolated MCP tool sessions, then grades the terminal backend state and side effects. Each task is executed 20 independent times from identical clean backends. The benchmark reports pass@1 (share of all attempts that succeeded), pass@20 (share of tasks solved at least once), and observed 20/20 (tasks that passed all 20 attempts). It also measures cost per successful task attempt.

## Takeaways

- ThinkingBox grades agents on the database state they leave behind, not on tool calls or final responses, revealing that 67.24% of failed attempts still terminated cleanly without error.
- A model’s single-attempt score does not predict its consistency; Kimi-K3 solved 93.89% of tasks at least once but only 13.41% on all 20 attempts, while Claude Opus 5 completed 47.53% of tasks on every attempt.
- The benchmark measures cost per successful task attempt, showing that consistency costs more than headline accuracy might suggest, with Claude Opus 5.5 costing $0.27 per successful attempt.

## FAQ

### How does ThinkingBox evaluate agents differently?

It grades agents on the terminal backend state and side effects left behind after MCP tool sessions, not on tool calls or final responses. This catches failures where agents terminate cleanly but leave wrong values or extra effects.

### Why does ThinkingBox run each task 20 times?

To measure consistency, not just capability. A single success does not indicate reliability; running 20 times reveals whether a model can perform correctly every time, reported as observed 20/20.

### How can I run ThinkingBox myself?

Through Hugging Face's OpenEnv. You need to install the ThinkingBox CLI, start Typesense and MCP servers, then use the \`thinkingbox-eval\` command with a YAML config naming your models.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Benchmark](https://yomu.fyi/topic/benchmark), [Evaluation](https://yomu.fyi/topic/evaluation), [LLMs](https://yomu.fyi/topic/llm)

- Source: [Hugging Face](https://huggingface.co/blog/microsoft/thinkingbox)
- Source URL: https://huggingface.co/blog/microsoft/thinkingbox
- Ingested by Yomu: 2026-10-04T00:00:13.821Z

[Read original post](https://huggingface.co/blog/microsoft/thinkingbox)