---
title: "Coding-Agent Benchmarks Should Match Their Users’ Task Flows"
description: "Coding-agent benchmarks like SWE-Bench Pro focus almost exclusively on single-step bug fixes and feature additions, while real developer use includes planning, code review, and questions. JetBrains Research analyzed 4,782 IDE sessions and found that less than half of developer requests are bugs and features, and tasks often arrive incrementally across multiple messages. SWE-Bench Pro tasks, however, are delivered as a single polished message, missing the interactive flow of actual usage. To bridge this gap, the authors propose SWE-TaskFlow, which rewrites existing issue-based tasks into multiple ordered requests and adds repository questions. Pilot experiments with GPT and Gemini models show that splitting tasks increases cost by 2.0-2.6 times but barely affects resolve rates, demonstrating that progressive disclosure primarily changes effort."
---

# Coding-Agent Benchmarks Should Match Their Users’ Task Flows

[Jetbrains](https://yomu.fyi/company/jetbrains) · Igor Slinko · Oct 20, 2026

**Type:** Problem & solution

## Summary

Coding-agent benchmarks like SWE-Bench Pro focus almost exclusively on single-step bug fixes and feature additions, while real developer use includes planning, code review, and questions. JetBrains Research analyzed 4,782 IDE sessions and found that less than half of developer requests are bugs and features, and tasks often arrive incrementally across multiple messages. SWE-Bench Pro tasks, however, are delivered as a single polished message, missing the interactive flow of actual usage. To bridge this gap, the authors propose SWE-TaskFlow, which rewrites existing issue-based tasks into multiple ordered requests and adds repository questions. Pilot experiments with GPT and Gemini models show that splitting tasks increases cost by 2.0-2.6 times but barely affects resolve rates, demonstrating that progressive disclosure primarily changes effort.

## Context

Current coding-agent benchmarks, especially issue-derived ones like SWE-Bench Pro, measure performance on tasks that arrive as single, complete requests for bug fixes or features. However, analysis of 4,782 real developer-agent sessions reveals a different pattern: tasks involve diverse intents (explain, plan, review, execute) and often unfold over multiple messages. Half of sessions are single-message, but the other 33% contain three or more messages, with intents like planning and code review appearing alongside bugs and features. This mismatch suggests benchmarks should match the task flows of their specific user populations.

## Approach / What changed

SWE-TaskFlow transforms issue-based benchmarks by using an LLM to rewrite each original issue into three sequential requests, preserving all original requirements and adding no new ones. A second pass verifies completeness. Additionally, repository questions are inserted at positions determined by the measured Task Flow (TFAS). The agent receives requests one by one, must answer questions before proceeding, and is scored on final patch correctness. This approach reuses existing repository snapshots, environments, and tests, adjusting only the user interaction layer.

## Takeaways

- Real developer-agent sessions involve diverse request types—bug fixes, features, explanations, planning, code reviews, and command execution—with only 45% being bugs and features combined. SWE-Bench Pro tasks are 94% bugs and features and arrive as single messages, missing this variety and interactivity.
- SWE-TaskFlow increases benchmark realism by splitting each issue into three ordered requests and inserting repository questions, simulating progressive disclosure. Pilot runs show this raises task cost 2.0-2.6x but only slightly reduces resolve rates (4-5 points), meaning the change primarily strains agent effort rather than capability.
- No single benchmark fits all user populations; SWE-TaskFlow provides a method to calibrate any issue-derived benchmark to a specific population’s measured task flow, including rare but critical tasks (e.g., destructive commands) that should be evaluated in separate stress-test slices.

## FAQ

### Why don't current benchmarks match real coding-agent use?

Real developers frequently ask agents to explain code, plan changes, review work, run commands, and debug—not just fix bugs or add features. These requests come in multiple messages across a session, not as one complete specification. SWE-Bench Pro tasks are almost entirely one-shot feature additions and bug fixes, ignoring the interactive, multi-intent nature of actual use.

### How does SWE-TaskFlow change the benchmark interaction?

SWE-TaskFlow reuses the original repository, tests, and requirements of issue-based benchmarks but rewrites each issue into three sequential messages using an LLM. It also inserts repository questions at natural points. The agent must handle progressive disclosure, answering questions and adapting its plan as new information arrives, making the task flow resemble real IDE sessions.

### What was the effect of splitting tasks in pilot experiments?

Splitting each issue into three sequential requests increased per-task cost by 2.0-2.6x across GPT and Gemini models. Resolve rates fluctuated by 4-5 points, similar to run-to-run variation, indicating that progressive disclosure mainly impacts effort rather than correctness. Adding repository questions raised cost by an additional 4-24%.

**Tags:** [Benchmark](https://yomu.fyi/topic/benchmark), [Coding Agents](https://yomu.fyi/topic/coding-agents), [SWE-Bench](https://yomu.fyi/topic/swe-bench), [Task Flows](https://yomu.fyi/topic/task-flows)

- Source: [Jetbrains](https://blog.jetbrains.com/research/2026/10/user-task-flow-paper)
- Source URL: https://blog.jetbrains.com/research/2026/10/user-task-flow-paper
- Ingested by Yomu: 2026-10-20T15:00:11.663Z

[Read original post](https://blog.jetbrains.com/research/2026/10/user-task-flow-paper)