---
title: "Measuring What Matters with Jules"
description: "Proactive coding agents need evaluation beyond task completion because existing public benchmarks such as SWE-Bench do not measure goal-oriented insight policies: deciding what matters, supporting evidence, and whether to interrupt or stay silent. To create preliminary ground truth, researchers clustered 705 bugs and 1,178 CLs from internal Google codebases using temporal proximity and semantic similarity, reverted each codebase to its pre-fix state, and gave the agent up to three exploration rounds. An LLM scored predicted insights from 1 to 5 against clustered bug targets, while success used average top score and Hit@K. With one exploration round, the agent averaged 4.5 out of 5; increasing the budget from two to three rounds raised Hit@5 from 33% to 57%, although the authors describe the findings as preliminary and are expanding the evaluation to public GitHub data and richer context streams."
---

# Measuring What Matters with Jules

[Google](https://yomu.fyi/company/google) · Nghi Bui, Georgios Evangelopoulos, Zack Elliott · Jun 22, 2026

**Type:** Benchmark

## Summary

Proactive coding agents need evaluation beyond task completion because existing public benchmarks such as SWE-Bench do not measure goal-oriented insight policies: deciding what matters, supporting evidence, and whether to interrupt or stay silent. To create preliminary ground truth, researchers clustered 705 bugs and 1,178 CLs from internal Google codebases using temporal proximity and semantic similarity, reverted each codebase to its pre-fix state, and gave the agent up to three exploration rounds. An LLM scored predicted insights from 1 to 5 against clustered bug targets, while success used average top score and Hit@K. With one exploration round, the agent averaged 4.5 out of 5; increasing the budget from two to three rounds raised Hit@5 from 33% to 57%, although the authors describe the findings as preliminary and are expanding the evaluation to public GitHub data and richer context streams.

## Context

Public benchmarks such as SWE-Bench measure completion of narrowly defined tasks, but the source says no benchmarks currently exist for higher-level goals. Evaluating proactive agents requires ground truth for judging their insight policies: what matters, what evidence supports it, and whether the agent should notify, question, draft, or stay silent.

## Approach / What changed

The evaluation clustered historical bugs using temporal proximity and semantic similarity, treating related bugs as evidence of broader aspirational goals. It used 705 bugs and 1,178 CLs from internal Google codebases, restored codebases to their pre-fix states, allowed up to three exploration rounds, and used an LLM to score predicted insights against ground-truth targets from 1 to 5.

## Takeaways

- The proposed evaluation treats individual bugs within related historical clusters as ground-truth targets for assessing whether an agent discovers a broader engineering goal.
- In the preliminary test, a single exploration round produced an average insight score of 4.5 out of 5 for straightforward engineering problems.
- Increasing the exploration budget from two to three rounds raised Hit@5 accuracy from 33% to 57%, indicating that additional passes helped uncover secondary signals.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [LLMs](https://yomu.fyi/topic/llm), [Model Evaluation](https://yomu.fyi/topic/model-evaluation)

- Source: [Google](https://developers.googleblog.com/measuring-what-matters-with-jules)
- Source URL: https://developers.googleblog.com/measuring-what-matters-with-jules
- Ingested by Yomu: 2026-09-09T14:29:28.126Z

[Read original post](https://developers.googleblog.com/measuring-what-matters-with-jules)