Loading…
How to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations
SalesforceScott Nyberg
Summary
Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.
Context
Traditional LLM evaluation relies on generated conversational output, which fails for production AI agents that use tool calling. Models can generate text claiming an action succeeded even when tool calls were omitted, execution sequences were violated, or external systems were left unchanged or corrupted.
Approach / What changed
CRMAgentBench evaluates multi-turn workflows in shared, stateful environments using all-or-nothing assertions that check tool selection, argument accuracy, execution sequence, final system state, and the absence of collateral damage or forbidden tools. System reliability across multiple attempts is measured using pass^k metrics.
Takeaways
- Conversational evaluations can register false successes because LLMs may narrate that a task was completed without actually executing the necessary tool calls.
- Strict all-or-nothing evaluation requires verifying correct tool selection, argument precision, call sequence, final system state, and the absence of forbidden actions or collateral changes.
- Pass^k metrics evaluate whether an agent consistently succeeds across k independent runs, distinguishing dependable models from high-variance models that pass@k metrics reward.
Related reading
How Agentforce-Powered AI Security Workflows Accelerate Incident Response
Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance.
Scott NybergRemoving the Security Barrier to Agentforce Adoption
Enterprise AI adoption across regulated industries stalls when internal governance mandates private, dedicated network paths for sensitive data. To eliminate the need for manual VPN tunnels and complex proxy layers, Salesforce re-engineered Private Connect into a multi-cloud networking platform connecting customer environments to Hyperforce services. The Private Connect v2.0 architecture utilizes private links, transit gateways, and direct endpoint routing alongside an automated control plane for rapid Hyperforce regional rollouts. In addition to supporting cross-substrate interconnects for Azure and decoupling endpoint management, the platform incorporates a generic connector framework supporting tools like Snowflake, Databricks, and Kafka. Today, the system provisions secure connectivity in under 30 minutes while processing roughly 120 TB of data and 683 million requests monthly across 15 AWS regions.
Scott NybergFrom Prediction to Action: How to Turn AI Outputs Into Decisions
Salesforce addressed an operational challenge where sellers faced roughly 12,000 dashboards and over 20 applications outputting machine learning predictions without clear next steps. The engineering team reframed machine learning outputs as raw signals rather than standalone answers. To bridge the gap between assessment and action, they built a Next Best Action layer that combines model signals, business logic, and contextual institutional knowledge into actionable recommendations. They integrated this layer with an AI agent using Model Context Protocol (MCP) tool contracts, enabling dynamic discovery and explicit handling of missing data. Finally, rather than introducing a separate dashboard destination, the agent serves on-demand recommendations directly inside Slack where sellers already collaborate.
Scott NybergHow AI Rebuilt Salesforce’s Decades-Old Localization Pipeline
Salesforce needed to deliver major releases across 34 languages as product localization volume surged by more than 35% under fixed budgets and unchanged delivery windows. To address these scaling constraints, the engineering team replaced their legacy translation workflows with an LLM-powered orchestration pipeline integrating prompt engineering, context engineering, and multi-stage validation. The architecture transforms existing style guides, glossaries, and terminology databases into structured context, routing interface strings through specialized prompt stages—up to 85 stages for some languages—to enforce product context, grammar, and branding rules. Combined with an automated validation loop spanning AI editing, AI validation, and human review, this system reduced localization costs by 50 to 90% while accelerating turnaround times across enterprise releases.
Scott Nyberg