# How AI observability works with MLflow

[Red Hat](https://yomu.fyi/company/red-hat) · Cedric Clyburn · Aug 26, 2026

**Type:** Explainer

## Summary

Traditional application monitoring treats AI workflows as black boxes, leaving engineering teams unable to determine whether incorrect responses result from model hallucinations or erroneous upstream tool data. MLflow provides AI observability by capturing distributed traces containing model calls, prompts, retrieved context, execution latencies, and tool interactions. In an example mortgage lending application, trace inspection revealed that an AI assistant accurately summarized raw JSON data returned by an internal pipeline tool rather than hallucinating conflicting metrics. Beyond latency attribution and tool debugging, MLflow is compatible with OpenTelemetry and provides automatic tracing integrations for frameworks such as LangChain and Claude Code. Capturing these traces enables teams to evaluate subsequent application versions, detect model provider shifts, and identify tool regressions before reaching users.

## Context

Traditional application monitoring treats AI workflows as black boxes, verifying only basic API success and overall latency. When an AI assistant produces metrics that conflict with dashboard data, teams cannot determine whether the language model hallucinated or if upstream tools supplied contradictory inputs.

## Approach / What changed

MLflow captures distributed traces across agentic queries, recording model calls, tool parameters, prompt contexts, token counts, and execution latencies. Inspecting span details within MLflow reveals the exact JSON payloads exchanged between the agent and tools like ceo\_pipeline\_summary, isolating whether errors stem from model synthesis or external tool outputs.

## Takeaways

- A single agentic query can generate over 30 metrics, such as token counts and LLM latency, alongside more than 15 log events within one distributed trace.
- Span inspection in MLflow allows developers to verify raw JSON inputs and outputs, distinguishing language model hallucination from upstream tool data discrepancies.
- MLflow is open source, compatible with OpenTelemetry, and supports automatic tracing integrations for AI frameworks including LangChain and Claude Code.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Open Source](https://yomu.fyi/topic/open-source)

[Read original post](https://developers.redhat.com/articles/2026/08/26/how-ai-observability-works-mlflow)