SalesforceShelby HeineckeAnnouncement
6 AI Agent Capabilities to Evaluate Before Enterprise Deployment
Summary
Salesforce’s AI Research team built EGI-Bench to evaluate whether an AI agent is truly ready for enterprise deployment. The framework combines eight independently developed enterprise benchmarks into a single scoring system organized around six skill dimensions: task outcomes, tool actioning, planning, grounding, safety, and engineering. EGI-Bench spans more than 20 subdomains and more than 1,500 test tasks, and it provides both a deep mode for comprehensive evaluation and a quick mode for a faster directional signal. Testing 14 LLMs revealed that no single model excelled across all dimensions, that multi-step planning remains a challenge, and that cost-efficient models are increasingly competitive. The framework is available on GitHub with customizable weights so practitioners can prioritize the capabilities most relevant to their use case.
Context
Salesforce’s AI Research team, led by Director Shelby Heinecke, built EGI-Bench to address the problem that many AI benchmarks assess narrow capabilities, but enterprise agents need a broad range of skills. An agent can pass narrow evaluations and still fail in production when it encounters real-world enterprise work. The framework spans more than 20 subdomains, more than 1,500 test tasks, and six enterprise skill dimensions.
Approach / What changed
The team combined eight independently developed enterprise benchmarks into a single framework, EGI-Bench. They broke each benchmark into its underlying sub-metrics, mapped those to six enterprise skill dimensions (task outcomes, tool actioning, planning, grounding, safety, engineering), normalized the scores so they could be meaningfully combined, and created a unified scoring system with a global EGI score and six subscores. Practitioners can run either a deep mode for comprehensive evaluation or a quick mode that subsamples each benchmark for a faster directional signal.
Takeaways
EGI-Bench evaluates enterprise agents across six skill dimensions: task outcomes, tool actioning, planning, grounding, safety, and engineering.
No single LLM was best across all six dimensions; different models have different strengths, so practitioners should choose a model based on the work their agent performs.
Cost-efficient models such as Haiku 4.5, GPT-5 mini, and Gemini 2.5 Flash delivered increasingly competitive performance across enterprise tasks.
FAQ
How does EGI-Bench make evaluation practical despite long run times?
EGI-Bench provides both deep and quick modes. Deep mode runs the benchmarks fully for a comprehensive evaluation, while quick mode subsamples each benchmark to produce a faster directional signal. If quick mode reveals a weakness in one area, practitioners can investigate that area more deeply rather than running every available test.
How were eight different benchmarks combined into one scoring system?
The team broke the eight benchmarks into their underlying sub-metrics and mapped those to the six enterprise dimensions. Because the metrics measure different things, the scores were normalized so they could be meaningfully combined. That produced six EGI-Bench subscores, which were then weighted to create a single global EGI score.
What did testing 14 LLMs reveal about model selection for enterprise agents?
No single model was best across all six dimensions. Multi-step planning remains challenging, with planning scores on the lower end across evaluated models. Cost-efficient models such as Haiku 4.5, GPT-5 mini, and Gemini 2.5 Flash delivered increasingly competitive performance across enterprise tasks.
Source and freshness
- Source
- engineering.salesforce.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.