# Ground truth is a process, not a dataset

[Amazon](https://yomu.fyi/company/amazon) · Venkatesh Saligrama · Jun 3, 2026

**Type:** Problem & solution

## Summary

Evaluating search-augmented AI research reports requires verifying complex, multi-source claims that depend heavily on surrounding context. Unassisted PhD-level domain specialists tasked with establishing static ground-truth labels achieved only 60.8% accuracy on a hidden test set during controlled testing. To resolve benchmark errors on cognitively demanding tasks, Amazon AGI introduced the audit-then-score evaluation protocol alongside DeepFact-Bench and the DeepFact-Eval verification system. Under this protocol, model disagreements serve as challenges requiring concrete counterevidence and written rationales, enabling human auditors to evaluate competing claims and update benchmark ground truth prior to scoring. Operating under this iterative workflow, expert test accuracy rose to 90.9% across four rounds, and DeepFact-Eval reached 83.4% accuracy with GPT-4.1.

## Context

Evaluating AI-generated deep research reports is difficult because individual claims often combine evidence across multiple sources and context. Traditional static benchmarks rely on human experts to establish fixed ground truth, but verifying complex literature-based claims creates high cognitive load, resulting in unassisted PhD-level experts achieving only 60.8% accuracy on a hidden test set of known answers.

## Approach / What changed

Amazon AGI introduced the audit-then-score protocol, DeepFact-Bench, and DeepFact-Eval. When a model disagrees with a benchmark label, it submits counterevidence and a written rationale as a challenger. Human auditors then compare the challenger's evidence directly against the original benchmark rationale to update the ground truth before final scoring.

## Takeaways

- Unassisted PhD-level specialists evaluating AI research claims from scratch achieved only 60.8% accuracy on a hidden set of known answers.
- Transitioning domain experts from one-shot labelers to auditors comparing competing rationales increased hidden test set accuracy from 60.8% to 90.9% across four rounds.
- DeepFact-Eval using GPT-4.1 achieved 83.4% accuracy on DeepFact-Bench, outperforming traditional fact checkers at 58.5% and a prior deep-research system at 69.1%.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://www.amazon.science/blog/ground-truth-is-a-process-not-a-dataset)