IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

Summary

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.

Takeaways

1. 1

   FM-3.3, Incorrect Verification, is reported as the strongest predictor of failure across the evaluated models; agents often declare success without checking ground truth.

2. 2

   Failure complexity rises across the cohort: Gemini-3-Flash averages 2.6 failure modes per failed trace, Kimi-K2 4.7, and GPT-OSS-120B 5.3, with the latter showing cascading failures.

3. 3

   The proposed interventions are model-sensitive: use hard tool evidence for Gemini, deterministic termination controls for Kimi-K2, and context hygiene with early error detection for GPT-OSS-120B.