---
title: "From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding"
description: "Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth."
---

# From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding

[Jetbrains](https://yomu.fyi/company/jetbrains) · Dmitry Savelev · Aug 31, 2026

**Type:** Benchmark

## Summary

Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth.

## Context

Resolve rate shows whether a coding agent solved a task but does not explain how it worked, what the run cost, or why failures occurred. The evaluation addresses the resulting gap by examining trajectories and patches alongside functional outcomes, especially because identical scores can conceal differences in efficiency, repository grounding, validation, and patch scope.

## Approach / What changed

The pipeline combines the issue, repository context, generated patch, test result, and execution trace. It uses deterministic measurements such as tokens, runtime, tool calls, modified symbols, repeated reads, retries, and complexity changes, then applies LLM judges to structured evidence to assess localization, root-cause identification, implementation, unnecessary complexity, and validation.

## Takeaways

- In a private benchmark, Claude Opus 4.7 and Gemini 3.5 Flash solved the same number of tasks, but Opus averaged 184 steps and USD 2.79 per run while Gemini averaged 271 steps and USD 1.24.
- The pipeline reports more than test outcomes by measuring efficiency, patch quality, and process quality, including repeated work, tool failure loops, repository grounding, and whether the final change was validated.
- Across the wider comparison, GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen 3.6 27B FP8 resolved fewer tasks at substantially lower cost. Profile conclusions are specific to the Junie scaffold.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Jetbrains](https://blog.jetbrains.com/junie/2026/08/from-leaderboards-to-profiles)
- Source URL: https://blog.jetbrains.com/junie/2026/08/from-leaderboards-to-profiles
- Ingested by Yomu: 2026-08-31T12:00:49.120Z

[Read original post](https://blog.jetbrains.com/junie/2026/08/from-leaderboards-to-profiles)