Together AIBenchmark
On DeepSWE, Kimi K3 trails Claude Fable 5 by 1.4 points at pass@1 (68.5% vs 69.9%) but leads at pass@2 (82.0 vs 80.2) and pass@4 (89.4% vs 88.5%) across 452 rollouts. Kimi K3 costs $4.65 per rollout versus $13.41 for Fable, delivering 2.8x more solved tasks per dollar. It covers 89.4% of the benchmark, solving 101 of 113 tasks, while Fable solves 100, though Fable is more reliable with 58 tasks solved four-for-four versus Kimi’s 45. The models share the highest cross-vendor similarity (0.72) and fail on the same tasks, with Kimi leading in Go and Fable leading in Python, JavaScript, TypeScript, and Rust. As an open-weight model, Kimi K3 offers a cost-effective alternative for retry-tolerant workloads.
Comparison of Kimi K3 and Claude Fable 5 on the DeepSWE benchmark, analyzing pass@1, pass@k, coverage, reliability, cost per rollout, and language-specific performance.
The authors ran 452 rollouts on DeepSWE for each model, with four trials per task graded by a hidden test suite. They compared pass@1, pass@2, pass@4, coverage, reliability, cost, and per-language scores.
Kimi K3 matches Claude Fable 5 on quality but costs a third as much per solved task, delivering 14.7 solves per $100 versus Fable’s 5.3.
Kimi K3 wins pass@2 (82.0 vs 80.2) and pass@4 (89.4% vs 88.5%), while Fable wins pass@1 (69.9% vs 68.5%) and four-for-four reliability (58 vs 45 tasks).
The models have the highest cross-vendor similarity (0.72) and succeed and fail on nearly the same tasks, so pairing them adds little diversity.
It depends on the metric. Claude Fable 5 wins single-attempt reliability on DeepSWE (pass@1 69.9% vs 68.5%) and solves more tasks four-for-four. Kimi K3 wins pass@2 and pass@4 and costs far less, so it is the stronger value pick for high volume or retry-tolerant agent work.
In our run, Kimi K3 cost $4.65 per rollout versus $13.41 for Claude Fable 5 at its xhigh setting, roughly a third of the price. Measured per solved task, Kimi K3 returned 14.7 solves per $100 against Fable's 5.3, about 2.8x the work per dollar.
Yes. Kimi K3 is an open-weight model from Moonshot AI, so once the weights are released it can be self-hosted or served through inference providers. Claude Fable 5 is a closed model available only through Anthropic and its partners.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
