How AI observability works with MLflow

Summary

Traditional application monitoring treats AI workflows as black boxes, leaving engineering teams unable to determine whether incorrect responses result from model hallucinations or erroneous upstream tool data. MLflow provides AI observability by capturing distributed traces containing model calls, prompts, retrieved context, execution latencies, and tool interactions. In an example mortgage lending application, trace inspection revealed that an AI assistant accurately summarized raw JSON data returned by an internal pipeline tool rather than hallucinating conflicting metrics. Beyond latency attribution and tool debugging, MLflow is compatible with OpenTelemetry and provides automatic tracing integrations for frameworks such as LangChain and Claude Code. Capturing these traces enables teams to evaluate subsequent application versions, detect model provider shifts, and identify tool regressions before reaching users.

Context

Traditional application monitoring treats AI workflows as black boxes, verifying only basic API success and overall latency. When an AI assistant produces metrics that conflict with dashboard data, teams cannot determine whether the language model hallucinated or if upstream tools supplied contradictory inputs.

Approach / What changed

MLflow captures distributed traces across agentic queries, recording model calls, tool parameters, prompt contexts, token counts, and execution latencies. Inspecting span details within MLflow reveals the exact JSON payloads exchanged between the agent and tools like ceo_pipeline_summary, isolating whether errors stem from model synthesis or external tool outputs.

Takeaways

• A single agentic query can generate over 30 metrics, such as token counts and LLM latency, alongside more than 15 log events within one distributed trace.
• Span inspection in MLflow allows developers to verify raw JSON inputs and outputs, distinguishing language model hallucination from upstream tool data discrepancies.
• MLflow is open source, compatible with OpenTelemetry, and supports automatic tracing integrations for AI frameworks including LangChain and Claude Code.

Related reading

• Red Hat · Aug 24, 2026

  Run LoRA fine-tuning on Red Hat OpenShift AI with Ray

  Red Hat OpenShift AI 3.5 integrates Ray cluster runtime images with preinstalled Training Hub packages to support distributed fine-tuning algorithms including LoRA, SFT, OSFT, and GRPO. Using the CodeFlare software development kit within a Jupyter workbench, developers can provision an elastic single-GPU Ray cluster attached to shared ReadWriteMany persistent storage containing model weights and training datasets. The tutorial demonstrates fine-tuning the Qwen2.5-1.5B-Instruct model on the public sql-create-context dataset to produce direct, bare SQL output without explanatory conversational text. After training completes, the workbench loads and merges the resulting LoRA adapter checkpoints with the base model to evaluate output format correctness against sample database schema questions. The RayCluster automatically tears down resources once the job concludes, while the shared pipeline architecture allows switching between supported fine-tuning algorithms by modifying dataset inputs and configuration parameters.

  Fiona Waters

• Red Hat ·