JetbrainsCiara ByrneExplainer
Scaling AI Agents: What We Learned From 9 Real-World Cases
Summary
A new report from JetBrains examines nine real-world cases of scaling AI agents in software development, revealing that adoption rates alone do not guarantee improved team performance. One organization reached 95% AI adoption without seeing a change in developer velocity, while another tripled pull request throughput and reduced its defect backlog by 54%. The report finds that effective scaling requires making development environments agent-ready, establishing shared standards, and investing in team learning. Fin, for example, created a dedicated team to manage agent infrastructure and invested in linting and standardization to handle increased output. The key lesson is that organizations must change the systems and practices around agents, not simply deploy more of them.
Context
Engineering leaders and practitioners from multiple organizations contributed case studies and accounts of scaling AI agents in software development. The report examines how teams organize work, maintain code quality, share knowledge, manage costs, and measure results when relying on AI agents.
Approach / What changed
The report synthesizes findings from nine case studies and practitioner accounts, including from Fin, Semrush, Skiller Whale, and JetBrains, to identify patterns in scaling AI agents. It focuses on changes to development workflows, codebase readiness, team learning, and measurement beyond adoption rates.
Takeaways
Fin tripled PR throughput and reduced its defect backlog by 54% over 16 months by investing in agent infrastructure, linting, and standardization, not just deploying more agents.
A codebase needs standardized build tools, fast tests, accessible APIs, and self-setup environments for agents to work reliably; fixing these gaps matters more than adding agents.
One organization with 95% AI adoption saw no velocity improvement until it shifted focus from encouraging AI use to helping developers delegate meaningful work and share effective practices.
FAQ
How did Fin manage to triple PR throughput?
Fin established a dedicated team, Team 2x, to manage infrastructure around agents, including shared skills, plugins, MCP servers, and evaluations. They tested agent skills against real examples, backtested code-review skills against historical PRs, and analyzed agent sessions to identify recurring mistakes and missing context.
What makes a codebase ready for AI agents?
Standardized build tools, fast tests, accessible APIs, and environments that agents can set up themselves are practical requirements. The lesson is to look at what prevents existing agents from completing tasks reliably before investing in more agents.
Why doesn't high AI adoption automatically improve team performance?
In one organization, AI adoption reached 95% yet developer velocity remained unchanged. The team shifted focus from encouraging AI use to helping developers delegate meaningful work, and champions who built richer repository-level context eventually opened PRs at three to four times their previous pace.
Source and freshness
- Source
- blog.jetbrains.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.