Loading post…
Together AIBenchmark
DeepSeek V4 Pro 0813 vs Claude Fable 5 on DeepSWE: Cost, Coding, and Routing
Summary
The DeepSWE benchmark comparison between DeepSeek V4 Pro 0813 and Claude Fable 5 reveals a trade-off between cost and accuracy. Claude Fable 5 leads on the first attempt with 69.7% pass@1 versus DeepSeek V4 Pro’s 62.8%, but costs 90 times more per rollout ($21.63 vs $0.24). Under repeated attempts, DeepSeek V4 Pro surpasses Fable, achieving 88.5% pass@4 compared to Fable’s 84.1%. The two models show unusually low per-task correlation (0.39), meaning they fail on different tasks. A cascade strategy that runs DeepSeek V4 Pro first and escalates to Claude Fable 5 only on failures solves 82.7% of tasks at $8.28 per task, outperforming either model used alone.
Context
Comparing DeepSeek V4 Pro 0813 and Claude Fable 5 on the DeepSWE benchmark to evaluate cost, coding accuracy, and routing strategy.
Approach / What changed
Run DeepSeek V4 Pro 0813 first and escalate to Claude Fable 5 only when the test suite rejects the answer. This cascade solves 82.7% of DeepSWE tasks at $8.28 per task, beating both models used individually.
Takeaways
Claude Fable 5 achieves 69.7% pass@1 on DeepSWE, but costs $21.63 per rollout – 90 times more than DeepSeek V4 Pro’s $0.24 per rollout.
DeepSeek V4 Pro 0813 matches Claude Fable 5 at pass@2 and exceeds it at pass@4, reaching 88.5% compared to Fable’s 84.1%.
The optimal routing strategy is a cascade: run DeepSeek V4 Pro first, escalate to Claude Fable 5 on failure, solving 82.7% of tasks at $8.28 per task – better and cheaper than using either model alone.
FAQ
Which model is more cost-effective per task on DeepSWE?
DeepSeek V4 Pro 0813 is far more cost-effective, solving 260 tasks per $100 compared to Claude Fable 5's 3 tasks. At $0.24 per rollout, Pro is 90x cheaper than Fable's $21.63.
What routing strategy yields the best accuracy at lowest cost?
A cascade that runs DeepSeek V4 Pro first and escalates to Claude Fable 5 only when the test suite rejects achieves 82.7% accuracy at $8.28 per task, more accurate than either model alone and cheaper than using Fable exclusively.
How do the models compare on Rust tasks?
Claude Fable 5 achieves 85% on Rust, a 20-point margin over DeepSeek V4 Pro's 65%, making it the clear choice for Rust-heavy workloads despite its high cost.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.