Loading post…
CloudflareRyan SkidmoreProblem & solution
Cutting our inference bill by 68% by being obsessive about optimization
Summary
By August 2026, Cloudflare’s AI code reviewer average cost had doubled to $2.46 per review, driven by a switch to Anthropic models and tripled token usage per session. Analysis showed 77% of spend went to prompt caching, yet first-request cache hits were only 14-26% because the shared prefix lacked an independent cache breakpoint. Adding a breakpoint on September 16 enabled cross-repository cache reuse, immediately cutting cost from $2.47 to $2.02. Subsequent changes—inline reference indices, tool removal, parallel file reads, and skipping reviewers on re-reviews—reduced cost by another $0.20 per review. The final result: $0.77 average cost (68% reduction), 1m32s median time (from 4m9s), and code quality at high effort maintained real-problem detection with reduced noise.
Context
The AI code reviewer’s cost doubled over five months due to model capability gains and a switch to Anthropic models, increasing token usage and session lengths. The team needed to cut costs without sacrificing review quality.
Approach / What changed
The team analyzed token usage and identified that first-request cache misses were the main cost driver. They added a cache breakpoint to the shared part of every agent prompt, ensuring the prefix was separately cached. They then moved reference indices into prompts to eliminate tool calls, removed unused tools to reduce token count and wasted calls, parallelized file reads, and skipped certain reviewers on re-reviews. These code changes accounted for about 40% of savings; the rest came from experimenting with different models and effort levels.
Takeaways
First-request cache hit rates for the AI reviewer were only 14-26% because the shared prefix was not cached independently; adding a cache breakpoint to the shared system prompt immediately reduced costs by 18%.
Moving reference indices inline in prompts eliminated three initial tool calls, reducing LLM calls per session from 15.9 to 9.6 and cutting cost per run from $1.12 to $0.72.
Removing unused tools from sub-reviewer contexts prevented wasted calls that occurred in 57-76% of sessions, reducing token count and call count per review.
FAQ
Why did first-request cache hits start so low?
Because OpenCode v1 placed only one cache breakpoint at the end of the combined system-prompt block containing both shared and repository-specific content, so the shared part was never stored independently. A review from one repository could not reuse a cached prefix from another.
How much of the savings came from code changes vs model selection?
About 40% of the savings came from code changes (prompt structuring, removing tools, etc.), while the rest came from experimenting with different models and effort levels.
Did lower cost affect review quality?
At high effort, code quality found as many real problems as before with less noise and fewer inflated severities. Medium effort lost some real findings, and security may have too, so both are being tested further at high effort.
Source and freshness
- Source
- blog.cloudflare.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.