---
title: "Benchmark"
description: "22 posts about Benchmark, summarised, each linking to the original."
---

# Benchmark
> 22 posts about Benchmark, summarised, each linking to the original.

## Articles

### [DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing](https://yomu.fyi/post/deepseek-v4-pro-0813-vs-gpt-5-6-sol-on-deepswe-cost-coding-and-routing.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 18, 2026
- Type: Benchmark

DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.


### [Kimi K3 vs Claude Fable 5 on DeepSWE: Cost and Coding](https://yomu.fyi/post/kimi-k3-vs-claude-fable-5-on-deepswe-cost-and-coding.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 24, 2026
- Type: Benchmark

On DeepSWE, Kimi K3 trails Claude Fable 5 by 1.4 points at pass@1 (68.5% vs 69.9%) but leads at pass@2 (82.0 vs 80.2) and pass@4 (89.4% vs 88.5%) across 452 rollouts. Kimi K3 costs $4.65 per rollout versus $13.41 for Fable, delivering 2.8x more solved tasks per dollar. It covers 89.4% of the benchmark, solving 101 of 113 tasks, while Fable solves 100, though Fable is more reliable with 58 tasks solved four-for-four versus Kimi’s 45. The models share the highest cross-vendor similarity (0.72) and fail on the same tasks, with Kimi leading in Go and Fable leading in Python, JavaScript, TypeScript, and Rust. As an open-weight model, Kimi K3 offers a cost-effective alternative for retry-tolerant workloads.


[Newer posts](https://yomu.fyi/topic/benchmark.md)