Loading post…
Hugging FaceAvijit Ghosh, Jenny Chim, Deep Joshi, Srishti, Matt Kennedy, Irene Solaiman, Jessica McFadyen, Lynn Tan, CozAnnouncement
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Summary
The UK AI Security Institute (AISI) has begun using EvalEval’s Every Eval Ever (EEE) schema and Evaluation Cards platform to openly share LLM evaluation results. This collaboration, stemming from a joint workshop at NeurIPS 2025, aims to make benchmark results more reproducible by providing standardized configuration and context alongside verified scores. AISI released results for five benchmarks—HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0—across six frontier models, plus two cyber evaluations using a partially overlapping model set. The release accompanies a paper studying how inference-time compute and evaluation protocol affect performance, revealing that models continued solving tasks when given correctness feedback. By adopting a shared reporting schema, evaluators can compare findings across studies and understand how setup choices influence reported performance, while EvalEval invites model developers, evaluation creators, and researchers to contribute to the shared infrastructure.
Context
The reproducibility of AI evaluation results is hindered by inconsistent reporting formats and insufficient setup information. Running costly evaluations again is often prohibitive. EvalEval and AISI collaborate to address this through shared infrastructure for standardized evaluation reporting.
Approach / What changed
AISI adopted EvalEval’s Every Eval Ever schema and Evaluation Cards platform to publicly share verified evaluation results with standardized configuration and context information, including data for five benchmarks and six frontier models.
Takeaways
AISI released verified evaluation results for HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0 covering Claude Opus 4, 4.5, 4.6, GPT-5, 5.2, and 5.4 models.
The paper shows that when models received correctness feedback from an oracle after each attempt, they continued to solve additional tasks as token use increased.
EvalEval invites model developers, evaluation developers, and researchers to report results or use the Every Eval Ever schema and Evaluation Cards to standardize evaluation reporting.
FAQ
What benchmarks and models are included in AISI's evaluation release?
The release includes HealthBench, FrontierMath, Humanity's Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0, covering Claude Opus 4, 4.5, 4.6, GPT-5, 5.2, and 5.4 models.
How does performance on Humanity's Last Exam change with inference compute?
When models received correctness feedback from an oracle after each attempt, they continued to solve additional tasks as token use increased, showing that protocol and compute affect performance.
Who is invited to contribute to the shared evaluation mission?
Model developers can report verified results, evaluation developers can use the Every Eval Ever schema, and researchers can explore Evaluation Cards to examine the state of evaluation reporting.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.