---
title: "How we eliminated $1 million a year of wasted AI agent spend in one hour"
description: "Databricks engineers investigated whether silent failures in MCP tool calls were driving hidden AI-agent costs as usage increased. Unity Gateway emitted OpenTelemetry traces for every invocation, recording tool names, arguments, errors, token counts, latency, and session IDs in a table that Genie One could query with plain-English questions. In a 24-hour sample, the analysis identified seven Jira and Google Drive/Docs bugs responsible for an estimated $499,000 annually in wasted tokens and 12,023 hours of agent wait time, or $1.2 million in lost productivity. The most costly patterns included under-specified interfaces, incompatible argument shapes, and cryptic or misleading errors that triggered retries and recovery loops. Engineers fixed the issues with a coding agent, completing the find-quantify-fix cycle in about an hour; the post recommends designing tools to tolerate reasonable LLM inputs and continuously tracing failures."
---

# How we eliminated $1 million a year of wasted AI agent spend in one hour

[Databricks](https://yomu.fyi/company/databricks) · Alkis Polyzotis · Sep 1, 2026

**Type:** Explainer

## Summary

Databricks engineers investigated whether silent failures in MCP tool calls were driving hidden AI-agent costs as usage increased. Unity Gateway emitted OpenTelemetry traces for every invocation, recording tool names, arguments, errors, token counts, latency, and session IDs in a table that Genie One could query with plain-English questions. In a 24-hour sample, the analysis identified seven Jira and Google Drive/Docs bugs responsible for an estimated $499,000 annually in wasted tokens and 12,023 hours of agent wait time, or $1.2 million in lost productivity. The most costly patterns included under-specified interfaces, incompatible argument shapes, and cryptic or misleading errors that triggered retries and recovery loops. Engineers fixed the issues with a coding agent, completing the find-quantify-fix cycle in about an hour; the post recommends designing tools to tolerate reasonable LLM inputs and continuously tracing failures.

## Context

As AI-agent usage expanded, aggregate token dashboards did not expose tool-call failures; retries and workarounds could look like ordinary usage growth while consuming tokens and developer time.

## Approach / What changed

The team used Unity Gateway's automatic OpenTelemetry traces for MCP invocations and Genie One to query the resulting trace table in plain English. They ranked recurring errors by frequency, recovery turns, token cost, and wait time, then applied fixes across the tool servers with a coding agent.

## Takeaways

- Unity Gateway's traces include tool name, arguments, errors, token counts, latency, and session ID for every MCP invocation, with records stored in one table.
- Jira's list/split bug failed 535 times daily, cost an estimated $87,000 annually in tokens and 4,850 hours of agent wait time, and took 12 turns to recover on average.
- Error-message quality tracked recovery behavior: cryptic tracebacks had a 30.5% repeat rate and 12.1 average recovery turns, versus 14% and 4.6 for self-documenting messages.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Genie](https://yomu.fyi/topic/genie), [MCP](https://yomu.fyi/topic/mcp), [OpenTelemetry](https://yomu.fyi/topic/opentelemetry)

- Source: [Databricks](https://www.databricks.com/blog/how-we-eliminated-1-million-year-wasted-ai-agent-spend-one-hour)
- Source URL: https://www.databricks.com/blog/how-we-eliminated-1-million-year-wasted-ai-agent-spend-one-hour
- Ingested by Yomu: 2026-09-02T08:00:52.694Z

[Read original post](https://www.databricks.com/blog/how-we-eliminated-1-million-year-wasted-ai-agent-spend-one-hour)