Red HatHema Veeradhi, Surya PathakTutorial
Reliable AI agents can fail in production through incorrect tool use, off-topic reasoning, or plausible but incorrect answers that unit tests, static benchmarks, and logs may not expose. This tutorial builds an eval-driven trace-evaluation pipeline on Red Hat OpenShift AI 3.4 or later, using MLflow to store structured agent traces and evaluation artifacts, IBM CLEAR to score reasoning quality, relevance, and tool usage with an LLM judge, and EvalHub to orchestrate OpenShift jobs. It uploads eight prerecorded multistep research-agent traces, each containing at least 30 spans, then verifies them in MLflow before deploying EvalHub, registering the ibm-clear provider, and configuring judge-model credentials. Standard mode evaluates steps holistically, while SPARC mode gives tool calls separate scoring rows; the resulting workflow produces critiques and ranked failure patterns with frequency, severity, and step or agent attribution.
AI agents can exhibit production failures that unit tests, static benchmarks, and logs do not fully reveal, including incorrect tool use, off-topic reasoning, and plausible but incorrect answers. Trace evaluation is used to analyze each execution step and identify recurring failure patterns, their frequency, severity, and origin.
The pipeline uses MLflow to capture and store structured agent traces and evaluation artifacts, IBM CLEAR for LLM-as-a-judge error analysis, and EvalHub to submit and orchestrate CLEAR evaluation jobs as OpenShift jobs. It demonstrates both holistic Standard mode and SPARC mode, which separates tool calls for finer-grained scoring.
MLflow stores agent execution traces containing LLM calls, tool invocations, node transitions, inputs, and outputs; it also stores CLEAR evaluation scores and artifacts.
IBM CLEAR generates critiques for reasoning quality, relevance, and tool usage, then clusters them into ranked failure patterns with frequency, severity, and step or agent attribution.
Standard mode scores each agent step holistically, whereas SPARC mode places tool calls in separate scoring rows to distinguish tool-use correctness from reasoning quality.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
AIOps automates the detect-investigate-analyze-recommend-execute-verify cycle by combining AI reasoning with infrastructure automation, keeping humans in control of credentials and final decisions.
AgentOps is the operational discipline of building, deploying, and monitoring production AI agents. Learn how tracing, evaluation, and governance turn raw agent behavior into reliable, cost-controlled systems.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
