---
title: "Catching poor LLM performance and accuracy before deployment"
description: "Rapid changes in vLLM and the path from upstream releases to Red Hat AI Inference images can introduce performance and accuracy regressions that ordinary CI/CD tests miss. The midstream regression detector benchmarks upstream and release-candidate builds across hardware variants, using GuideLLM at multiple concurrency levels for performance and GSM8K for accuracy. It stores nightly upstream results, compares them with midstream benchmarks against threshold deltas, and feeds confirmed failures into fixes and rebuilds before release. To reduce false positives from limited samples, the workflow adds Wasserstein distance as a secondary check of whole request-response distributions rather than relying only on medians or means. The system also connects benchmark runs with prior JIRA regressions through a custom Claude skill that suggests possible fixes and assesses whether a flagged change is genuine."
---

# Catching poor LLM performance and accuracy before deployment

[Red Hat](https://yomu.fyi/company/red-hat) · Christopher Miyai · Sep 17, 2026

**Type:** Benchmark

## Summary

Rapid changes in vLLM and the path from upstream releases to Red Hat AI Inference images can introduce performance and accuracy regressions that ordinary CI/CD tests miss. The midstream regression detector benchmarks upstream and release-candidate builds across hardware variants, using GuideLLM at multiple concurrency levels for performance and GSM8K for accuracy. It stores nightly upstream results, compares them with midstream benchmarks against threshold deltas, and feeds confirmed failures into fixes and rebuilds before release. To reduce false positives from limited samples, the workflow adds Wasserstein distance as a secondary check of whole request-response distributions rather than relying only on medians or means. The system also connects benchmark runs with prior JIRA regressions through a custom Claude skill that suggests possible fixes and assesses whether a flagged change is genuine.

## Context

Red Hat AI Inference packages vLLM, llm-d, and vLLM's llm-compressor into production images for hardware variants including NVIDIA CUDA, AMD ROCm, and Google TPU. Differences in dependencies, base operating systems, and hardware can silently affect model performance between upstream and downstream builds.

## Approach / What changed

A nightly scanner finds new upstream vLLM releases, benchmarks listed models on different hardware, and stores the results in cloud storage. Release candidates are benchmarked after midstream changes, then compared with the stored upstream results using threshold deltas. Performance uses GuideLLM across concurrency levels, while accuracy uses GSM8K; Wasserstein distance supplements point statistics to assess distribution-wide shifts, and a custom Claude skill uses benchmark data and prior JIRA regressions for triage.

## Takeaways

- The detector compares upstream and midstream model benchmarks before release, covering performance metrics such as time to first token, throughput, and intertoken latency, plus accuracy on GSM8K.
- A fixed 5% threshold can produce false positives for median or mean statistics when request counts are limited; Wasserstein distance provides a secondary check of the overall response distribution.
- Benchmark regressions can depend on the downstream dependency set, base OS, or hardware image. The workflow supports diagnosis, rebuilding with a fix, and rerunning tests until the release is ready.

## FAQ

### What does the detector compare before a Red Hat AI Inference release?

It compares benchmarks from an upstream vLLM release with benchmarks from the midstream release candidate. The comparison covers models across hardware variants and checks performance and accuracy against configured threshold deltas.

### Why add Wasserstein distance instead of relying only on a 5% threshold?

A fixed threshold applied to point statistics such as the median or mean can create false positives, especially with limited requests. Wasserstein distance checks the holistic shift of the request distribution, while remaining a secondary check rather than replacing point statistics.

### How are flagged regressions investigated?

The team identifies the specific issue, rebuilds with a fix, and runs the benchmark workflow again. A custom Claude skill also connects benchmark runs with prior JIRA regression tickets to suggest fixes and estimate whether a flagged regression is genuine.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/17/catching-poor-llm-performance-and-accuracy-before-deployment)
- Source URL: https://developers.redhat.com/articles/2026/09/17/catching-poor-llm-performance-and-accuracy-before-deployment
- Ingested by Yomu: 2026-09-17T14:06:17.586Z

[Read original post](https://developers.redhat.com/articles/2026/09/17/catching-poor-llm-performance-and-accuracy-before-deployment)