Driving the Agent Quality Flywheel from Your Coding Agent

Summary

The quality-flywheel skill addresses the gap between prompt tweaks that improve a few examples and evidence that an agent improved without regressions. A coding agent orchestrates dataset preparation, inference, trace grading, failure analysis, and optimization, using Google's adaptive AutoRaters or custom metrics while keeping the optimizer and evaluator independent. In a travel-concierge example, the revision_honored rubric found that 21% of cases ignored mid-conversation trip changes, exceeding the stated action threshold. A separate software-bug-assistant cycle raised tool-disclosure responses from 0% to 96% across 15 cases, while production traces and online monitors extend the same loop beyond synthetic scenarios. Human approval remains required, and adaptive scores are treated as directional signals rather than absolute grades.

Takeaways

1. 1

   The workflow runs Prepare Data, Run Inference, Grade, Analyze Failures, and Optimize & Iterate; grading always runs, while inference is skipped when complete traces already exist.

2. 2

   The optimizer never grades its own work: fixes are proposed by a coding agent or other optimizer, then scored independently by the Gemini Enterprise Agent Platform GenAI evaluation service.

3. 3

   Adaptive AutoRaters provide broad directional signals, while stable custom rubrics or counts can measure a specific behavior across iterations; synthetic scenarios bootstrap evaluation, but production traces sharpen the loop.