Loading post…
Together AIBenchmark
Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
Summary
GPT-5.6 Sol and Kimi K3 were compared on the DeepSWE software engineering benchmark using 904 graded rollouts across 113 tasks. Sol wins single-shot pass@1 (72.7% vs 68.5%) and reliability (84.5% tasks solved four-for-four versus 76.6%), while Kimi K3 leads pass@4 (89.4% vs 85.8%) with wider coverage at a substantially lower cost—$4.65 per rollout versus Sol’s $8.37, delivering 14.7 solves per $100 against Sol’s 5.3. The two models have a low task correlation of 0.46 and disagree on 18 tasks, enabling a cascade routing strategy that runs Kimi first and escalates to Sol when the test suite rejects the result, achieving 85.6% accuracy, beating both models individually and a perfect one-shot oracle router. Kimi K3 is open-weight and available on Together AI.
Context
The post evaluates two large language models on the DeepSWE software engineering benchmark, comparing single-shot and multi-shot accuracy, cost, and the benefits of routing between them.
Approach / What changed
A cascade routing strategy where Kimi K3 is attempted first; if its output fails the test suite, the task is escalated to GPT-5.6 Sol. This approach yields 85.6% accuracy at a cost lower than using Sol alone.
Takeaways
On DeepSWE, GPT-5.6 Sol achieves a higher single-shot pass@1 (72.7%) than Kimi K3 (68.5%), but Kimi K3 wins on pass@4 (89.4% vs 85.8%) due to wider coverage across multiple attempts.
Kimi K3 costs $4.65 per rollout versus Sol’s $8.37, delivering 2.8 times more solved tasks per dollar, making it the stronger value for high-volume or retry-tolerant agent work.
The two models have low task-level correlation (0.46), so routing a Kimi-first cascade that escalates to Sol on test failure achieves 85.6% accuracy, outperforming either model alone and cheaper per solve than Sol.
FAQ
Is Kimi K3 better than GPT-5.6 Sol?
It depends on the metric. GPT-5.6 Sol wins single-attempt quality with pass@1 72.7% vs 68.5% and solves more tasks four-for-four. Kimi K3 wins pass@2 and pass@4 and costs far less, so it is the stronger value for high-volume or retry-tolerant work. For most teams, routing between them is best.
How much cheaper is Kimi K3 than GPT-5.6 Sol?
Kimi K3 costs $4.65 per rollout versus Sol's $8.37, a little over half the price. Per solved task, Kimi K3 returns 14.7 solves per $100 against Sol's 5.3, about 2.8 times the work per dollar.
How high can routing between them get you?
With a Kimi-first cascade that uses a test suite as verifier, routing achieves about 85.6% accuracy on DeepSWE. This beats either model alone (Sol's 72.7% or Kimi's 68.5% pass@1) and even a perfect one-shot oracle router (83.4%), because the cascade gives hard tasks two independent attempts.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.