Why don’t machine learning research agents overfit?

Summary

Repeatedly optimizing against long-lived machine-learning benchmarks appears to invite overfitting, yet the post notes that gains on heavily reused benchmarks have largely transferred to newly created test sets. It describes a paper that resets LLM-based research agents, gives an explorer repeated access to validation feedback, and passes the resulting strategy through a token bottleneck to a fresh reproducer. Across eight datasets, strategies from hundreds of experiments were reproducible from prompts as short as 16 to 32 tokens; at shorter budgets, performance diverged, showing that the compressed message carried data-dependent choices rather than only prior knowledge. Deliberate overfitting occurred in 38 of 102 runs, with validation accuracy more than 10% above held-out accuracy, but those gains disappeared after compression; possible pretraining side channels remain unresolved.

Takeaways

1. 1

   Across eight datasets, strategies produced after hundreds of experiments could be reproduced from prompts as short as 16 to 32 tokens.

2. 2

   Reducing explorer feedback to one bit per query still produced strategies as good as those found with full numerical validation scores.

3. 3

   In 38 of 102 deliberately overfitting runs, validation accuracy exceeded held-out accuracy by more than 10%; those validation-specific gains disappeared after prompt compression.