Topic

SWE-Bench

1 post about SWE-Bench. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Jetbrains
Coding-Agent Benchmarks Should Match Their Users’ Task Flows
Coding-agent benchmarks like SWE-Bench Pro focus almost exclusively on single-step bug fixes and feature additions, while real developer use includes planning, code review, and questions. JetBrains Research analyzed 4,782 IDE sessions and found that less than half of developer requests are bugs and features, and tasks often arrive incrementally across multiple messages. SWE-Bench Pro tasks, however, are delivered as a single polished message, missing the interactive flow of actual usage. To bridge this gap, the authors propose SWE-TaskFlow, which rewrites existing issue-based tasks into multiple ordered requests and adds repository questions. Pilot experiments with GPT and Gemini models show that splitting tasks increases cost by 2.0-2.6 times but barely affects resolve rates, demonstrating that progressive disclosure primarily changes effort.

The best engineering reads, once a week.

Topic

1 post about SWE-Bench. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Jetbrains
Coding-Agent Benchmarks Should Match Their Users’ Task Flows
Coding-agent benchmarks like SWE-Bench Pro focus almost exclusively on single-step bug fixes and feature additions, while real developer use includes planning, code review, and questions. JetBrains Research analyzed 4,782 IDE sessions and found that less than half of developer requests are bugs and features, and tasks often arrive incrementally across multiple messages. SWE-Bench Pro tasks, however, are delivered as a single polished message, missing the interactive flow of actual usage. To bridge this gap, the authors propose SWE-TaskFlow, which rewrites existing issue-based tasks into multiple ordered requests and adds repository questions. Pilot experiments with GPT and Gemini models show that splitting tasks increases cost by 2.0-2.6 times but barely affects resolve rates, demonstrating that progressive disclosure primarily changes effort.

The best engineering reads, once a week.