The Agent Said It Was Done. The Database Disagreed.

Summary

ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data.

Takeaways

1. 1

   ThinkingBox grades agents on the database state they leave behind, not on tool calls or final responses, revealing that 67.24% of failed attempts still terminated cleanly without error.

2. 2

   A model’s single-attempt score does not predict its consistency; Kimi-K3 solved 93.89% of tasks at least once but only 13.41% on all 20 attempts, while Claude Opus 5 completed 47.53% of tasks on every attempt.

3. 3

   The benchmark measures cost per successful task attempt, showing that consistency costs more than headline accuracy might suggest, with Claude Opus 5.5 costing $0.27 per successful attempt.

FAQ