Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase

Summary

Databricks built a coding-agent benchmark from reviewed tasks drawn from its multi-million-line codebase, spanning languages including Scala, Go, Rust, Python, and TypeScript. It uses recent human-written pull requests, held-out tests, standard model and harness setups, and manual validation rather than an LLM judge; Git history was sealed after traces showed agents could recover solutions. Results clustered models and harnesses into three capability tiers and showed that quality, cost per task, and token price can diverge: GLM 5.2 statistically tied Opus 4.8 on quality while costing $1.28 versus $1.94 per task, whereas Sonnet 5 cost $2.09 versus Opus’s $1.94. The same model through different harnesses produced cost differences of more than 2x in some cases while quality remained the same; Pi sent about three times less context per turn. Databricks plans to shift routine work toward cheaper models, deploy open models such as GLM as daily drivers, and improve model-and-harness selection.

Context

The expanding range of coding models and harnesses made it important to compare performance on real Databricks engineering tasks and understand how task performance varies with price. Public benchmarks did not represent Databricks’ multi-language codebase, and their public tasks could eventually leak into training data.

Approach / What changed

The benchmark was built from recent, human-written pull requests with high-quality tests and self-contained changes representative of the stack. Task prompts captured intent while omitting the original solution, relevant tests were held out, and each model-harness combination ran with standard tools. Results were evaluated by running the tests after checkpointing the agent’s completed code, with manual review and sealed Git history as additional safeguards.

Takeaways

• Models and harnesses formed three broad capability tiers: expensive top-tier models handled varied and difficult work, while less capable models remained effective for common tasks at lower cost.
• Token price did not predict end-to-end task cost. Sonnet 5 was about 1.7x cheaper per token than Opus 4.8 but cost more per task because it consumed 1.9x more tokens.
• Harness context management materially affected efficiency. Pi sent about three times less context per turn and sometimes reduced task cost by more than 2x without changing quality.

Related reading