---
title: "Evaluating LLM guardrail configs locally with EvalHub"
description: "This installment evaluates local NeMo Guardrails configurations against large-scale risk datasets using EvalHub, following earlier manual testing. EvalHub provides a unified interface for AI evaluation, with provider integrations, reusable benchmarks, Hugging Face dataset inputs, and both local and Kubernetes deployment options. The benchmark configuration maps dataset prompt and label columns to block and pass labels, while balanced safe and risky examples prevent an always-block guardrail from appearing perfect. On the prompt-injection benchmark, the nuanced regex configuration achieved 51.1% accuracy with 13.7 ms median latency; DeBERTa reached 88.93% accuracy with 45.2 ms median latency. The conclusion favors layered guardrails: targeted regex checks can provide speed, with DeBERTa handling other cases, and EvalHub supplies reproducible measurements of the latency–efficacy trade-off."
---

# Evaluating LLM guardrail configs locally with EvalHub

[Red Hat](https://yomu.fyi/company/red-hat) · Rob Geada · Sep 3, 2026

**Type:** Benchmark

## Summary

This installment evaluates local NeMo Guardrails configurations against large-scale risk datasets using EvalHub, following earlier manual testing. EvalHub provides a unified interface for AI evaluation, with provider integrations, reusable benchmarks, Hugging Face dataset inputs, and both local and Kubernetes deployment options. The benchmark configuration maps dataset prompt and label columns to block and pass labels, while balanced safe and risky examples prevent an always-block guardrail from appearing perfect. On the prompt-injection benchmark, the nuanced regex configuration achieved 51.1% accuracy with 13.7 ms median latency; DeBERTa reached 88.93% accuracy with 45.2 ms median latency. The conclusion favors layered guardrails: targeted regex checks can provide speed, with DeBERTa handling other cases, and EvalHub supplies reproducible measurements of the latency–efficacy trade-off.

## Context

Manual testing from the preceding part is extended to rigorous evaluation against popular large-scale risk datasets. The benchmark must include both allowed and blocked prompts because risk-only data can make an always-block guardrail report 100% accuracy while destroying functional utility.

## Approach / What changed

The evaluation uses EvalHub's NeMo Guardrails community provider and predefined prompt\_injection benchmark, with Hugging Face datasets mapped through prompt and label columns plus block and pass labels. Local setup launches the EvalHub server, then evalhub eval run compares nuanced regex and DeBERTa configurations; results are retrieved by job UUID.

## Takeaways

- Risk-only datasets can make a guardrail that blocks every prompt score 100% accuracy, so evaluations should include both safe and risky prompts.
- The nuanced regex guardrail achieved 51.1% accuracy and 13.7 ms median latency, while DeBERTa achieved 88.93% accuracy and 45.2 ms median latency.
- The post proposes layering targeted regex checks with the more accurate DeBERTa classifier to balance latency and guardrail efficacy.

**Tags:** [AI Security](https://yomu.fyi/topic/ai-security), [EvalHub](https://yomu.fyi/topic/evalhub), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [NeMo Guardrails](https://yomu.fyi/topic/nemo-guardrails)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/03/evaluating-llm-guardrail-configs-locally-with-evalhub)
- Source URL: https://developers.redhat.com/articles/2026/09/03/evaluating-llm-guardrail-configs-locally-with-evalhub
- Ingested by Yomu: 2026-09-03T20:00:51.546Z

[Read original post](https://developers.redhat.com/articles/2026/09/03/evaluating-llm-guardrail-configs-locally-with-evalhub)