JetbrainsIgor SlinkoProblem & solution
Coding-agent benchmarks like SWE-Bench Pro focus almost exclusively on single-step bug fixes and feature additions, while real developer use includes planning, code review, and questions. JetBrains Research analyzed 4,782 IDE sessions and found that less than half of developer requests are bugs and features, and tasks often arrive incrementally across multiple messages. SWE-Bench Pro tasks, however, are delivered as a single polished message, missing the interactive flow of actual usage. To bridge this gap, the authors propose SWE-TaskFlow, which rewrites existing issue-based tasks into multiple ordered requests and adds repository questions. Pilot experiments with GPT and Gemini models show that splitting tasks increases cost by 2.0-2.6 times but barely affects resolve rates, demonstrating that progressive disclosure primarily changes effort.
Current coding-agent benchmarks, especially issue-derived ones like SWE-Bench Pro, measure performance on tasks that arrive as single, complete requests for bug fixes or features. However, analysis of 4,782 real developer-agent sessions reveals a different pattern: tasks involve diverse intents (explain, plan, review, execute) and often unfold over multiple messages. Half of sessions are single-message, but the other 33% contain three or more messages, with intents like planning and code review appearing alongside bugs and features. This mismatch suggests benchmarks should match the task flows of their specific user populations.
SWE-TaskFlow transforms issue-based benchmarks by using an LLM to rewrite each original issue into three sequential requests, preserving all original requirements and adding no new ones. A second pass verifies completeness. Additionally, repository questions are inserted at positions determined by the measured Task Flow (TFAS). The agent receives requests one by one, must answer questions before proceeding, and is scored on final patch correctness. This approach reuses existing repository snapshots, environments, and tests, adjusting only the user interaction layer.
Real developer-agent sessions involve diverse request types—bug fixes, features, explanations, planning, code reviews, and command execution—with only 45% being bugs and features combined. SWE-Bench Pro tasks are 94% bugs and features and arrive as single messages, missing this variety and interactivity.
SWE-TaskFlow increases benchmark realism by splitting each issue into three ordered requests and inserting repository questions, simulating progressive disclosure. Pilot runs show this raises task cost 2.0-2.6x but only slightly reduces resolve rates (4-5 points), meaning the change primarily strains agent effort rather than capability.
No single benchmark fits all user populations; SWE-TaskFlow provides a method to calibrate any issue-derived benchmark to a specific population’s measured task flow, including rare but critical tasks (e.g., destructive commands) that should be evaluated in separate stress-test slices.
Real developers frequently ask agents to explain code, plan changes, review work, run commands, and debug—not just fix bugs or add features. These requests come in multiple messages across a session, not as one complete specification. SWE-Bench Pro tasks are almost entirely one-shot feature additions and bug fixes, ignoring the interactive, multi-intent nature of actual use.
SWE-TaskFlow reuses the original repository, tests, and requirements of issue-based benchmarks but rewrites each issue into three sequential messages using an LLM. It also inserts repository questions at natural points. The agent must handle progressive disclosure, answering questions and adapting its plan as new information arrives, making the task flow resemble real IDE sessions.
Splitting each issue into three sequential requests increased per-task cost by 2.0-2.6x across GPT and Gemini models. Resolve rates fluctuated by 4-5 points, similar to run-to-run variation, indicating that progressive disclosure mainly impacts effort rather than correctness. Adding repository questions raised cost by an additional 4-24%.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
An adversarial prefix is a challenge prepended to a prompt to see whether a model holds its original decision. This diagram maps the layers that turn that pressure test into a measurable, reviewable signal.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
