---
title: "Using AI to Build AI: How Agentforce Engineers Scale Automation Without Sacrificing Reliability"
description: "The Agentforce team at Salesforce faced the challenge of using AI to help build and maintain the Atlas Reasoning Engine without compromising reliability. The central risk was that AI could diagnose problems, modify the engine, test fixes, and review its own work while being confidently wrong. To address this, the team built an integrated engineering knowledge base using semantic retrieval to connect fragmented knowledge across incidents, fixes, and discussions. They implemented layered AI evaluation with an agent-as-a-judge model and human oversight, using different models to review work from different perspectives to avoid self-preference bias. As a result, typical bug-resolution time dropped from two to four weeks to one or two days, and system availability reached 100% in weekly reports during the observed period."
---

# Using AI to Build AI: How Agentforce Engineers Scale Automation Without Sacrificing Reliability

[Salesforce](https://yomu.fyi/company/salesforce) · Fereydoon Vafaei · Nov 9, 2026

**Type:** Problem & solution

## Summary

The Agentforce team at Salesforce faced the challenge of using AI to help build and maintain the Atlas Reasoning Engine without compromising reliability. The central risk was that AI could diagnose problems, modify the engine, test fixes, and review its own work while being confidently wrong. To address this, the team built an integrated engineering knowledge base using semantic retrieval to connect fragmented knowledge across incidents, fixes, and discussions. They implemented layered AI evaluation with an agent-as-a-judge model and human oversight, using different models to review work from different perspectives to avoid self-preference bias. As a result, typical bug-resolution time dropped from two to four weeks to one or two days, and system availability reached 100% in weekly reports during the observed period.

## Context

The Agentforce team at Salesforce needed to use AI to help build and maintain the Atlas Reasoning Engine, the brain of Agentforce, without allowing unreliable AI-generated work to compromise it. The central risk was that AI could diagnose a problem, modify the engine, test the fix, and review its own work while being confidently wrong. Fragmented engineering knowledge made recurring failures harder to diagnose and prevent.

## Approach / What changed

The team built an integrated engineering knowledge base using semantic retrieval to connect incidents with previous fixes, code, Slack discussions, and notes. They implemented layered AI evaluation with an agent-as-a-judge model where different models review work from different perspectives to avoid self-preference bias, combined with human oversight. They also use a risk-based autonomy model: agents design, implement, review, and deploy changes to non-production environments, but never merge to main or deploy to production without explicit human approval.

## Takeaways

- The team reduced bug-resolution time from two to four weeks to one or two days by using AI-assisted bug resolution with semantic knowledge retrieval that connects incidents with previous fixes and discussions.
- AI evaluation uses an agent-as-a-judge model where different models review work from different perspectives to avoid self-preference bias, evaluating the full trajectory of reasoning rather than just final output, with engineers overseeing and signing off.
- The team uses a risk-based autonomy model: agents can design, implement, review, and deploy changes to non-production environments, but never merge to main or deploy to production without explicit human approval.

## FAQ

### How did the team ensure AI evaluation was reliable?

They used layered evaluation with an agent-as-a-judge model where different models judge each other's work, evaluating the full trajectory of reasoning rather than just final output. Engineers oversee evaluations across many incidents and sign off on findings, preventing self-preference bias.

### What metrics proved AI automation improved engineering?

Bug-resolution time dropped from two to four weeks to one or two days. An agent performs approximately 10 automated code reviews at once. Knowledge retrieval is measured by precision and recall. System availability reached 100% in weekly reports during the observed period.

### How did the team handle the risk of AI autonomy?

They use a risk-based autonomy model: automation is applied aggressively when outcomes are verifiable, actions are bounded, and failures are detectable. Agents never merge to main or deploy to production without explicit human approval. Human oversight is preserved for high-risk decisions.

**Tags:** [Agentic AI](https://yomu.fyi/topic/agentic-ai), [AI](https://yomu.fyi/topic/ai), [Automation](https://yomu.fyi/topic/automation), [Reliability](https://yomu.fyi/topic/reliability), [Software Engineering](https://yomu.fyi/topic/software-engineering)

- Source: [Salesforce](https://engineering.salesforce.com/using-ai-to-build-ai-how-agentforce-engineers-scale-automation-without-sacrificing-reliability)
- Source URL: https://engineering.salesforce.com/using-ai-to-build-ai-how-agentforce-engineers-scale-automation-without-sacrificing-reliability
- Ingested by Yomu: 2026-11-09T21:00:12.303Z

[Read original post](https://engineering.salesforce.com/using-ai-to-build-ai-how-agentforce-engineers-scale-automation-without-sacrificing-reliability)
