Loading post…
Red HatChristopher MiyaiBenchmark
Catching poor LLM performance and accuracy before deployment
Summary
Rapid changes in vLLM and the path from upstream releases to Red Hat AI Inference images can introduce performance and accuracy regressions that ordinary CI/CD tests miss. The midstream regression detector benchmarks upstream and release-candidate builds across hardware variants, using GuideLLM at multiple concurrency levels for performance and GSM8K for accuracy. It stores nightly upstream results, compares them with midstream benchmarks against threshold deltas, and feeds confirmed failures into fixes and rebuilds before release. To reduce false positives from limited samples, the workflow adds Wasserstein distance as a secondary check of whole request-response distributions rather than relying only on medians or means. The system also connects benchmark runs with prior JIRA regressions through a custom Claude skill that suggests possible fixes and assesses whether a flagged change is genuine.
Context
Red Hat AI Inference packages vLLM, llm-d, and vLLM's llm-compressor into production images for hardware variants including NVIDIA CUDA, AMD ROCm, and Google TPU. Differences in dependencies, base operating systems, and hardware can silently affect model performance between upstream and downstream builds.
Approach / What changed
A nightly scanner finds new upstream vLLM releases, benchmarks listed models on different hardware, and stores the results in cloud storage. Release candidates are benchmarked after midstream changes, then compared with the stored upstream results using threshold deltas. Performance uses GuideLLM across concurrency levels, while accuracy uses GSM8K; Wasserstein distance supplements point statistics to assess distribution-wide shifts, and a custom Claude skill uses benchmark data and prior JIRA regressions for triage.
Takeaways
The detector compares upstream and midstream model benchmarks before release, covering performance metrics such as time to first token, throughput, and intertoken latency, plus accuracy on GSM8K.
A fixed 5% threshold can produce false positives for median or mean statistics when request counts are limited; Wasserstein distance provides a secondary check of the overall response distribution.
Benchmark regressions can depend on the downstream dependency set, base OS, or hardware image. The workflow supports diagnosis, rebuilding with a fix, and rerunning tests until the release is ready.
FAQ
What does the detector compare before a Red Hat AI Inference release?
It compares benchmarks from an upstream vLLM release with benchmarks from the midstream release candidate. The comparison covers models across hardware variants and checks performance and accuracy against configured threshold deltas.
Why add Wasserstein distance instead of relying only on a 5% threshold?
A fixed threshold applied to point statistics such as the median or mean can create false positives, especially with limited requests. Wasserstein distance checks the holistic shift of the request distribution, while remaining a secondary check rather than replacing point statistics.
How are flagged regressions investigated?
The team identifies the specific issue, rebuilds with a fix, and runs the benchmark workflow again. A custom Claude skill also connects benchmark runs with prior JIRA regression tickets to suggest fixes and estimate whether a flagged regression is genuine.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.