What We Learned by Reproducing 2,200 papers from ICML

Summary

An ICML 2026 reproduction hackathon examined whether coding agents could audit research at conference scale, amid 23,918 submissions and 6,352 accepted papers. From July 15 to August 2, 1,221 participants published 6,816 Trackio logbooks covering 2,226 papers and 35,908 claims; an open-weights GLM-5.2 judge assigned per-claim verdicts, while participants used cloud jobs and toy reproductions when full experiments were impossible. At least one claim was independently verified in 51% of examined papers, including 266 fully reproduced papers, while 23% had a falsified or contested claim and 242 papers produced opposing verdicts across teams. Adversarial re-checks confirmed issues including a paging-theorem error, a reverse-versus-forward-KL mismatch, and padded evaluation tokens, but also caught false falsifications, leading to the conclusion that agents expand auditing capacity while humans remain necessary to steer experiments and assess cases such as image quality.

Takeaways

1. 1

   Of 2,226 attempted papers, 266 were fully reproduced, 632 were partially reproduced with no falsified claims, 502 had toy-scale evidence only, and 280 remained inconclusive.

2. 2

   Reproducibility was not binary: 242 papers had independent teams reach opposite verdicts on the same claims, and 23% of examined papers had at least one falsified or contested claim.

3. 3

   The most reliable workflows involved human steering; agents encountered local loops, scale-dependent behavior, units mismatches, and evaluation questions requiring human perceptual judgment.