ThinkingBox evaluates agent performance by comparing the final database state with the expected outcome, catching agents that claim completion without correct database changes. Its benchmark, ThinkingBox-Bench, contains 507 tasks across domains like retail, auto insurance, travel, neobank, and consulting, each with executable checks on terminal database state. The authors used these executable outcomes as rewards for reinforcement learning, fine-tuning Qwen3.8-27B with GRPO for 50 steps on a training pool selected by offline policy-relative difficulty, which filters tasks that are neither already solved nor hopeless. After training, observed 20/20 tasks completed correctly on all twenty attempts rose from 38 (7.50%) to 107 (21.10%), while pass@1 increased from 51.70% to 60.78%, with a small decline in at-least-once coverage. The results show that database-state grading can serve as a reward signal to improve agent repeatability, and the training loop and data curation method are released on GitHub.
Context
This post follows up on the authors’ earlier work introducing ThinkingBox-Bench, which revealed that many agents achieve high pass@1 but low repeatability. Here they use the same executable database-state checks as rewards for post-training agents with GRPO.
Approach / What changed
The authors built a training pool from ThinkingBox-format tasks outside the benchmark, estimated each task’s success probability using the base policy before training, and kept the pool fixed. They fine-tuned Qwen3.8-27B with GRPO for 50 steps, using the executable terminal-state verdict as the reward. The training loop oversamples and discards zero-variance groups to avoid gradient problems.
Takeaways
1
ThinkingBox-Bench evaluates agents by checking the final database state rather than tool calls, incentivizing discovery of novel solution paths.
2
Offline policy-relative difficulty selects training tasks that are neither already solved nor hopeless for the current model, improving training efficiency.
3
After 50 GRPO steps, Qwen3.8-27B’s observed 20/20 tasks rose from 38 to 107, while pass@1 increased from 51.70% to 60.78%, with a slight drop in at-least-once coverage.
FAQ
How does ThinkingBox evaluate agent performance?
ThinkingBox compares the final database state with the expected outcome using executable checks. It does not rely on tool calls, so it catches agents that claim completion without correct database changes.
What is offline policy-relative difficulty?
It is a method to select training tasks by estimating each task's success probability using the base policy before training. Tasks that are already solved or hopeless are excluded, keeping only those that are challenging for the current model.
What metrics does ThinkingBox-Bench report?
It reports pass@1 (average success across attempts), pass@20 (tasks solved at least once), and observed 20/20 (tasks solved on all twenty attempts). The post emphasizes that observed 20/20 is closer to production reliability.
Cekura generates simulated caller scenarios to test voice-agent prompts, so failures surface before real calls. The trade-off: fast, repeatable iteration covers only the failures you thought to script.
A five-stage walkthrough of how CBRN risk evaluations are commissioned, designed, graded, and converted into safety-level decisions for frontier AI models.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.