---
title: "Pass@1 vs. pass@k: Two Ways to Read One Benchmark"
description: "A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload."
---

# Pass@1 vs. pass@k: Two Ways to Read One Benchmark

**Type:** [Comparison](https://yomu.fyi/visual-explainers/category/comparison.md)

![A two-column comparison diagram: the left column covers attempt-level scoring with pass@1 and per-rollout cost, the right column covers outcome-level scoring with pass@k and cost per solved task, joined by a lower band about harness and workload shape plus a decision cue bar.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/e942864e-0e47-420a-9f79-0cc1b4348ef4/image-6fe178cd-f7a3-4d73-97fa-394ed13eb1d1.webp)

> A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.

## Explanation

Benchmark results rarely settle a decision on their own, because the same run can be scored two ways. At the attempt level, pass@1 asks whether one try succeeds and reports cost per rollout; at the outcome level, pass@2 and pass@4 allow retries and shift the unit of value to cost per solved task. The two readings can even disagree, since a model that trails on first-attempt accuracy can lead once retries are permitted, and cheaper models can solve far more tasks per dollar. The explainer also shows why the harness matters: static request patterns and Poisson arrivals produce different latency distributions, and throughput claims only mean something against a stated workload shape.

## FAQ

### Why can two models tie at pass@1 and still differ at pass@4?

pass@1 scores a single attempt, so a gap inside the error bars is not decisive. On DeepSWE, GLM-5.3 and Claude Fable 5 tied near 69% at pass@1, yet GLM-5.3 led at pass@4, 87.6% versus 84.1%. Retries recover failures that a single attempt records as a loss, so the scoring rule can change the verdict.

### When is cost per solved task more useful than cost per rollout?

When your pipeline may retry. DeepSeek V4 Pro 0813 costs $0.24 per rollout and $100 buys 260 solved tasks versus 9 for GPT-5.6 Sol, which the per-rollout price hides. A cascade of cheap model first, escalate on test failure, solved 83.0% of tasks at $3.35 each, beating either model alone.

### Why does the benchmark harness change the latency numbers?

AIPerf shows static synthetic requests producing one latency distribution, while Poisson arrivals averaging 10 requests per second with variable lengths produce a wider spread. The load client must also be kept from becoming the bottleneck. Percentiles such as TTFT and inter-token latency describe the workload shape, not the model alone.

**Tags:** [Benchmark](https://yomu.fyi/topic/benchmark.md), [Cost Optimization](https://yomu.fyi/topic/cost-optimization.md), [Latency](https://yomu.fyi/topic/latency.md), [Model Evaluation](https://yomu.fyi/topic/model-evaluation.md), [Model Routing](https://yomu.fyi/topic/model-routing.md), [Performance](https://yomu.fyi/topic/performance.md)

## Related reading

### [Benchmarking LLM Inference at Scale with AIPerf](https://yomu.fyi/post/benchmarking-llm-inference-at-scale-with-aiperf.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 18, 2026
- Type: Benchmark

AIPerf is NVIDIA’s successor to GenAI-Perf, intended to benchmark large-scale LLM inference without letting the load client become the bottleneck. Its multiprocess architecture combines worker processes, separate record-processor services, and ZMQ coordination, while supporting 15+ endpoint types, public datasets, and trace-replay formats. The walkthrough serves Qwen3-0.6B through vLLM, using streaming synthetic requests with fixed 128-token inputs and outputs; AIPerf reports TTFT, ITL, request latency, output token throughput, percentile distributions, and optional GPU telemetry in CSV and JSON. A second profile uses Poisson arrivals averaging 10 requests per second, variable input and output lengths, and a fixed random seed, producing wider latency distributions than the static baseline as concurrency and workload variability increase.


### [Benchmark Shows MongoDB Atlas Manages Thousands of Payments Per Second with Zero Data Loss](https://yomu.fyi/post/benchmark-shows-mongodb-atlas-manages-thousands-of-payments-per-second.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Published: Sep 9, 2026
- Type: Benchmark

A benchmark of the Icon Payments Framework (IPF), trusted by Citi, NatWest, and BNP Paribas, on MongoDB Atlas demonstrated extreme performance and resilience for instant payment processing. The system sustained 6,000 end-to-end payments per second, requiring approximately 430,000 simultaneous database operations, with a mean end-to-end latency of 0.51 seconds and persistence latency below 30 milliseconds. Upgrading from MongoDB Atlas 8.0 to 8.3 reduced persistence latency by roughly 50%, from 40 to 20 milliseconds at 4,000 payments per second, and throughput scaled near-linearly across the 500 to 6,000 range while latency remained flat until peak and hot-shard CPU utilization sat at about half. During a primary failover, a new node was elected in five seconds with zero data loss and no application restart. An application node failure recovered to 3,500 transactions per second in 90 seconds with zero data loss.


### [GLM-5.3 vs. GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing](https://yomu.fyi/post/glm-5-3-vs-gpt-5-6-sol-on-deepswe-cost-coding-and-routing.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 21, 2026
- Type: Benchmark

GLM-5.3 and GPT-5.6 Sol were evaluated on 113 DeepSWE tasks with 904 rollouts, comparing accuracy, cost, and failure patterns. Sol edges ahead at pass@1 (72.7% vs 69.0%), but GLM-5.3 leads at pass@4 (87.6% vs 85.8%) at $3.99 per rollout, half Sol’s $8.37 cost and yielding 17 solves per $100 versus Sol’s 9. Their low per-task correlation of 0.43 makes a cascade effective: GLM-5.3 first, then Sol on test failure, achieving 85.9% solved at $6.61 per task. Sol runs faster (19 minutes, 61 steps vs 35 minutes, 124 steps) and achieves higher reliability (61 tasks solved all four tries vs 48), but it breaks passing tests in 20% of failures versus GLM-5.3’s 11%.


### [GLM-5.3 vs. Claude Fable 5 on DeepSWE: Cost, Coding, and Routing](https://yomu.fyi/post/glm-5-3-vs-claude-fable-5-on-deepswe-cost-coding-and-routing.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 21, 2026
- Type: Benchmark

GLM-5.3 and Claude Fable 5 were compared on the DeepSWE benchmark across 113 tasks with four trials each, totaling 904 rollouts. On first-attempt accuracy, Fable 5 scored 69.7% and GLM-5.3 scored 69.0%, a statistical tie inside error bars. GLM-5.3 led on pass@2 (81.1% vs. 77.1%) and pass@4 (87.6% vs. 84.1%), and cost $3.99 per rollout versus Fable’s $21.63, a 5.4x price gap. Per $100, GLM-5.3 solved 17 tasks while Fable solved 3. Both models regress test suites in only 11% of failures, and GLM-5.3 averages 35 minutes per rollout with 80k output tokens versus Fable’s 34 minutes and 114k tokens. The models are near-substitutes with 0.65 per-task correlation; GLM-5.3 won five of eight domains including JavaScript and concurrency, while Fable excelled only in Rust and serialization, making GLM-5.3 the practical default.


### [DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing](https://yomu.fyi/post/deepseek-v4-pro-0813-vs-gpt-5-6-sol-on-deepswe-cost-coding-and-routing.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 18, 2026
- Type: Benchmark

DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.


### [Kimi K3 vs Claude Fable 5 on DeepSWE: Cost and Coding](https://yomu.fyi/post/kimi-k3-vs-claude-fable-5-on-deepswe-cost-and-coding.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 24, 2026
- Type: Benchmark

On DeepSWE, Kimi K3 trails Claude Fable 5 by 1.4 points at pass@1 (68.5% vs 69.9%) but leads at pass@2 (82.0 vs 80.2) and pass@4 (89.4% vs 88.5%) across 452 rollouts. Kimi K3 costs $4.65 per rollout versus $13.41 for Fable, delivering 2.8x more solved tasks per dollar. It covers 89.4% of the benchmark, solving 101 of 113 tasks, while Fable solves 100, though Fable is more reliable with 58 tasks solved four-for-four versus Kimi’s 45. The models share the highest cross-vendor similarity (0.72) and fail on the same tasks, with Kimi leading in Go and Fable leading in Python, JavaScript, TypeScript, and Rust. As an open-weight model, Kimi K3 offers a cost-effective alternative for retry-tolerant workloads.