The Agentforce team at Salesforce faced the challenge of using AI to help build and maintain the Atlas Reasoning Engine without compromising reliability. The central risk was that AI could diagnose problems, modify the engine, test fixes, and review its own work while being confidently wrong. To address this, the team built an integrated engineering knowledge base using semantic retrieval to connect fragmented knowledge across incidents, fixes, and discussions. They implemented layered AI evaluation with an agent-as-a-judge model and human oversight, using different models to review work from different perspectives to avoid self-preference bias. As a result, typical bug-resolution time dropped from two to four weeks to one or two days, and system availability reached 100% in weekly reports during the observed period.
Context
The Agentforce team at Salesforce needed to use AI to help build and maintain the Atlas Reasoning Engine, the brain of Agentforce, without allowing unreliable AI-generated work to compromise it. The central risk was that AI could diagnose a problem, modify the engine, test the fix, and review its own work while being confidently wrong. Fragmented engineering knowledge made recurring failures harder to diagnose and prevent.
Approach / What changed
The team built an integrated engineering knowledge base using semantic retrieval to connect incidents with previous fixes, code, Slack discussions, and notes. They implemented layered AI evaluation with an agent-as-a-judge model where different models review work from different perspectives to avoid self-preference bias, combined with human oversight. They also use a risk-based autonomy model: agents design, implement, review, and deploy changes to non-production environments, but never merge to main or deploy to production without explicit human approval.
Takeaways
1
The team reduced bug-resolution time from two to four weeks to one or two days by using AI-assisted bug resolution with semantic knowledge retrieval that connects incidents with previous fixes and discussions.
2
AI evaluation uses an agent-as-a-judge model where different models review work from different perspectives to avoid self-preference bias, evaluating the full trajectory of reasoning rather than just final output, with engineers overseeing and signing off.
3
The team uses a risk-based autonomy model: agents can design, implement, review, and deploy changes to non-production environments, but never merge to main or deploy to production without explicit human approval.
FAQ
How did the team ensure AI evaluation was reliable?
They used layered evaluation with an agent-as-a-judge model where different models judge each other's work, evaluating the full trajectory of reasoning rather than just final output. Engineers oversee evaluations across many incidents and sign off on findings, preventing self-preference bias.
What metrics proved AI automation improved engineering?
Bug-resolution time dropped from two to four weeks to one or two days. An agent performs approximately 10 automated code reviews at once. Knowledge retrieval is measured by precision and recall. System availability reached 100% in weekly reports during the observed period.
How did the team handle the risk of AI autonomy?
They use a risk-based autonomy model: automation is applied aggressively when outcomes are verifiable, actions are bounded, and failures are detectable. Agents never merge to main or deploy to production without explicit human approval. Human oversight is preserved for high-risk decisions.
A five-step how-to for scaling AI adoption across engineering teams: federate ownership, ship one shared paved path, let builders own their tools, measure throughput, and feed bottlenecks back into the path.
Active learning is a loop that spends a fixed labeling budget on the samples most worth annotating. This diagram maps its stages — pool, score, label, train, evaluate — and the return path that closes the cycle.
Follow one contact through a call center: arrival, routing, handling by a human or AI agent, resolution, and the cost-per-contact math that decides which path to use.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.