BenchMIRT: What are LLM benchmarks actually measuring?

Summary

BenchMIRT is a method for auditing LLM benchmarks at the individual-prompt level, where questions may measure capabilities beyond a benchmark’s stated goal. It applies multidimensional Item Response Theory (MIRT) to model and question results, estimating model abilities plus each item’s difficulty and discrimination across dimensions. Trained on 100 LLMs, 16 benchmarks, and more than 34K questions, it independently recovered safety and general reasoning as dominant dimensions, repeatedly producing the same result. The analysis found, for example, that BBQ and WMDP aligned more strongly with reasoning than safety, while HarmBench combined safety- and reasoning-associated question groups; retaining 10% of items generally preserved capability rankings, and held-out correctness prediction reached 79% versus 70% for a baseline. BenchMIRT’s dimensions depend on the benchmark set, and its item-level transparency can also enable evaluation trimming.

Takeaways

1. 1

   BenchMIRT independently recovered safety and general reasoning as dominant dimensions from results across 100 LLMs, 16 benchmarks, and more than 34K questions.

2. 2

   BBQ and WMDP scores aligned more strongly with general reasoning than safety, while HarmBench contained question groups associated with both safety and general reasoning.

3. 3

   Keeping 10% of questions generally preserved the relative picture of model capability; held-out item correctness was predicted 79% of the time, compared with 70% for a simpler baseline.