Editing Files at 1000 Tokens per Second

Summary

Large code edits expose coding agents to laziness, inaccuracy, and latency, especially when frontier models must rewrite hundreds of lines. The post presents fast apply, a specialized full-file code-edit model that separates planning from applying: Cursor's frontier model plans the change, while the apply model rewrites the current file. It evaluates roughly 450 edits on files under 400 lines, using Claude-3 Opus as grader, and finds llama-3-70b-ft nearly matches claude-3-opus-diff while outperforming GPT-4-Turbo and GPT-4o. Rather than generate conventional diffs, it uses full-file rewrites and search/replace hunks, which avoid line-number errors and give models more output tokens for reasoning. A custom speculative decoding method, deployed with Fireworks, reaches approximately 1,000 tokens per second on a 70b model, about 13x faster than vanilla Llama-3-70b inference and 9x faster than the previous GPT-4 deployment.

Takeaways

1. 1

   The evaluation set contains roughly 450 full-file edits involving files under 400 lines; Claude-3 Opus is used as the grader, with scores that may bias toward Claude outputs.

2. 2

   The best finetuned model, llama-3-70b-ft, nearly matches claude-3-opus-diff and outperforms GPT-4-Turbo, GPT-4o, and GPT-4-turbo-spec on the reported evaluations.

3. 3

   Speculative edits reaches about 1,000 tokens per second on a 70b model, delivering roughly a 13x speedup over vanilla Llama-3-70b inference and a 9x speedup over the previous GPT-4 speculative-edits deployment.