How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Summary

Recent autonomous machine learning engineering agents increasingly rely on elaborate machinery such as multi-agent orchestrators and dedicated retrieval subagents, motivated by progress stagnation over long-horizon cycles and limited Large Language Model primitives. This paper finds that under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline that uses direct read, write, and bash primitives. The authors argue via large-scale systematic ablation studies that the machinery layers become redundant in the coding agent setting. The backbone, not the harness complexity, is the primary driver for performance. Consequently, the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

Takeaways

1. 1

   Under equal time budget and same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a minimal-harness coding agent baseline.

2. 2

   The machinery layers such as multi-agent orchestrators and retrieval subagents become redundant in the coding agent setting.

3. 3

   Effort spent on hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

FAQ