---
title: "How we compare model quality in Cursor"
description: "Cursor describes a hybrid online-offline process for comparing coding-agent quality as developer tasks become longer, more complex, and multi-file. Its offline CursorBench suite uses tasks derived from real Cursor sessions, paired with ground-truth solutions through Cursor Blame, and evaluates correctness alongside code quality, efficiency, and interaction behavior; the post focuses on correctness. Tasks are intentionally short and underspecified, span larger codebases and multi-workspace environments, and are scored with agentic graders, while correctness is plotted against median completion tokens to show the compute/latency trade-off. Controlled live-traffic experiments supplement the suite, catching regressions that offline grading misses, and the post reports that CursorBench separates frontier models more clearly and its rankings track developer experience more closely than public benchmarks."
---

# How we compare model quality in Cursor

[Cursor](https://yomu.fyi/company/cursor) · Naman Jain · Mar 11, 2026

**Type:** Benchmark

## Summary

Cursor describes a hybrid online-offline process for comparing coding-agent quality as developer tasks become longer, more complex, and multi-file. Its offline CursorBench suite uses tasks derived from real Cursor sessions, paired with ground-truth solutions through Cursor Blame, and evaluates correctness alongside code quality, efficiency, and interaction behavior; the post focuses on correctness. Tasks are intentionally short and underspecified, span larger codebases and multi-workspace environments, and are scored with agentic graders, while correctness is plotted against median completion tokens to show the compute/latency trade-off. Controlled live-traffic experiments supplement the suite, catching regressions that offline grading misses, and the post reports that CursorBench separates frontier models more clearly and its rankings track developer experience more closely than public benchmarks.

## Context

Developers are asking coding agents to handle longer, more complex work across multiple files, tools, and steps. The post argues that public benchmarks can be misaligned with developer workflows, restrict valid solutions through narrow grading, and suffer from contamination, making it harder to distinguish models with different practical utility.

## Approach / What changed

Cursor combines CursorBench, an offline suite built from real Cursor sessions and ground-truth solutions traced through Cursor Blame, with controlled online evaluations on live traffic. The suite uses short, underspecified tasks, agentic graders, and multiple performance dimensions; online experiments measure interaction and output signals and test specific changes through controlled ablations.

## Takeaways

- CursorBench tasks are refreshed every few months and have grown substantially in lines of code and mean file count, adding scenarios such as monorepos, production-log investigation, and long-running experiments.
- The evaluation plots correctness against median completion tokens, making the compute and latency trade-off visible rather than treating quality as the only measure.
- Removing semantic search in a controlled online ablation helped identify where the tool mattered most, including repository-grounded question answering on larger codebases.

**Tags:** [AI](https://yomu.fyi/topic/ai), [AI Agents](https://yomu.fyi/topic/ai-agents)

- Source: [Cursor](https://cursor.com/blog/cursorbench)
- Source URL: https://cursor.com/blog/cursorbench
- Ingested by Yomu: 2026-09-02T11:43:53.268Z

[Read original post](https://cursor.com/blog/cursorbench)