---
title: "Add NeMo Guardrails to a LangGraph agent on OpenShift AI"
description: "LangGraph agents can call tools and follow prompts but lack safety guardrails, risking unsafe responses like explaining fraud. NeMo Guardrails acts as a transparent proxy between the agent and LLM, inspecting every request and response without requiring agent code changes. On Red Hat OpenShift AI, the guardrails service is deployed via a NemoGuardrails custom resource managed by the TrustyAI Operator, and the agent’s BASE_URL points to this service instead of the model endpoint. The example demonstrates three rail types: regex filtering for jailbreak patterns, content safety classification against S1-S13 categories, and topic safety to keep the agent within a banking domain, with optional tracing using MLflow and OpenTelemetry. Rails execute sequentially, with the first failure short-circuiting the chain, and the separation lets safety policy be owned by guardrails while the agent handles reasoning and tools."
---

# Add NeMo Guardrails to a LangGraph agent on OpenShift AI

[Red Hat](https://yomu.fyi/company/red-hat) · Tarun Etikala · Oct 5, 2026

**Type:** Tutorial

## Summary

LangGraph agents can call tools and follow prompts but lack safety guardrails, risking unsafe responses like explaining fraud. NeMo Guardrails acts as a transparent proxy between the agent and LLM, inspecting every request and response without requiring agent code changes. On Red Hat OpenShift AI, the guardrails service is deployed via a NemoGuardrails custom resource managed by the TrustyAI Operator, and the agent’s BASE\_URL points to this service instead of the model endpoint. The example demonstrates three rail types: regex filtering for jailbreak patterns, content safety classification against S1-S13 categories, and topic safety to keep the agent within a banking domain, with optional tracing using MLflow and OpenTelemetry. Rails execute sequentially, with the first failure short-circuiting the chain, and the separation lets safety policy be owned by guardrails while the agent handles reasoning and tools.

## Context

Adding NeMo Guardrails as a proxy between a LangGraph agent and LLM on Red Hat OpenShift AI to enforce safety policies without modifying agent code.

## Approach / What changed

Deploy NeMo Guardrails as a proxy service using the TrustyAI Operator’s NemoGuardrails custom resource on OpenShift AI. Point the LangGraph agent’s BASE\_URL at the guardrails service. Configure three rail types (regex, content safety, topic safety) in config.yaml and prompts.yml. Optionally enable tracing with MLflow and OpenTelemetry.

## Takeaways

- NeMo Guardrails uses a proxy pattern that sits between the agent and LLM, inspecting requests and responses without requiring changes to the agent’s code.
- Rails execute sequentially; the first failure short-circuits the chain, so a blocked input rail prevents further processing and returns a configured refusal.
- On OpenShift AI, the guardrails service is deployed via a NemoGuardrails custom resource managed by the TrustyAI Operator, and the agent’s BASE\_URL is pointed at the guardrails service instead of the model endpoint.

## FAQ

### How does the proxy pattern work?

NeMo Guardrails sits between the agent and LLM, inspecting traffic in both directions. It passes system prompts and tool calls through unchanged but checks user messages and LLM responses against configured rails. Allowed requests reach the inference endpoint; blocked requests receive a configured refusal.

### What rail types are demonstrated?

The example includes three rail types: regex filtering for jailbreak patterns (no LLM call), content safety classification against S1-S13 categories (violence, criminal planning, profanity), and topic safety classification to keep the agent within a banking domain.

### How can I observe guardrail decisions?

Red Hat OpenShift AI provides two complementary views: agent-level tracing in MLflow shows user queries, tool calls, and responses, while rail-level tracing with OpenTelemetry shows per-rail spans, which rail blocked, and timing for each layer.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [LangGraph](https://yomu.fyi/topic/langgraph), [NeMo Guardrails](https://yomu.fyi/topic/nemo-guardrails), [OpenShift AI](https://yomu.fyi/topic/openshift-ai), [Safety](https://yomu.fyi/topic/safety)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/10/05/add-nemo-guardrails-langgraph-agent-openshift-ai)
- Source URL: https://developers.redhat.com/articles/2026/10/05/add-nemo-guardrails-langgraph-agent-openshift-ai
- Ingested by Yomu: 2026-10-05T15:00:09.145Z

[Read original post](https://developers.redhat.com/articles/2026/10/05/add-nemo-guardrails-langgraph-agent-openshift-ai)