---
title: "IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST"
description: "IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches."
---

# IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

[Hugging Face](https://yomu.fyi/company/hugging-face) · Ayhan Sebin, Rohan Arora, Saurabh Jha · Feb 18, 2026

**Type:** Benchmark

## Summary

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.

## Context

ITBench success rates show whether an agent failed but do not identify the underlying cause, leaving developers without a structured way to diagnose failures in high-stakes IT automation workflows.

## Approach / What changed

The analysis annotates 310 ITBench SRE execution traces with MAST, a taxonomy of 14 failure patterns across system design, inter-agent misalignment, and task verification. The traces represent Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B operating in realistic environments, and are assessed using mean recall.

## Takeaways

- FM-3.3, Incorrect Verification, is reported as the strongest predictor of failure across the evaluated models; agents often declare success without checking ground truth.
- Failure complexity rises across the cohort: Gemini-3-Flash averages 2.6 failure modes per failed trace, Kimi-K2 4.7, and GPT-OSS-120B 5.3, with the latter showing cascading failures.
- The proposed interventions are model-sensitive: use hard tool evidence for Gemini, deterministic termination controls for Kimi-K2, and context hygiene with early error detection for GPT-OSS-120B.

**Tags:** [Kubernetes](https://yomu.fyi/topic/kubernetes), [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Hugging Face](https://huggingface.co/blog/ibm-research/itbenchandmast)
- Source URL: https://huggingface.co/blog/ibm-research/itbenchandmast
- Ingested by Yomu: 2026-08-28T16:00:55.205Z

[Read original post](https://huggingface.co/blog/ibm-research/itbenchandmast)