---
title: "Run decision models on vLLM and Red Hat AI using DiffusionGemma"
description: "A new category of AI model called ‘System One’ makes fast, typed, probabilistic decisions from unstructured input, exemplified by TypeSafe AI’s Jev, which offers guaranteed structure, default probabilities, and low latency (70–500 ms) for high-volume decisions like routing and moderation. Jev is a hosted API without self-hosting options, prompting the vLLM community to adapt DiffusionGemma, an open block-diffusion model, as a self-hostable decision engine. The approach uses a fixed-length canvas seeded with answer templates, a single denoising step, and calibrated logprobs to return decisions with confidence scores, achieving over 300 decisions per second on a Cloud Run deployment. DiffusionGemma is already validated on Red Hat AI, and the structured-read support in vLLM nightly enables teams to run decision models on their own infrastructure. The post provides a recipe using vLLM nightly and a decision server that exposes a Jev-compatible endpoint for prototyping."
---

# Run decision models on vLLM and Red Hat AI using DiffusionGemma

[Red Hat](https://yomu.fyi/company/red-hat) · Lucas Wilkinson, Rob Greenberg · Sep 28, 2026

**Type:** Explainer

## Summary

A new category of AI model called ‘System One’ makes fast, typed, probabilistic decisions from unstructured input, exemplified by TypeSafe AI’s Jev, which offers guaranteed structure, default probabilities, and low latency (70–500 ms) for high-volume decisions like routing and moderation. Jev is a hosted API without self-hosting options, prompting the vLLM community to adapt DiffusionGemma, an open block-diffusion model, as a self-hostable decision engine. The approach uses a fixed-length canvas seeded with answer templates, a single denoising step, and calibrated logprobs to return decisions with confidence scores, achieving over 300 decisions per second on a Cloud Run deployment. DiffusionGemma is already validated on Red Hat AI, and the structured-read support in vLLM nightly enables teams to run decision models on their own infrastructure. The post provides a recipe using vLLM nightly and a decision server that exposes a Jev-compatible endpoint for prototyping.

## Context

TypeSafe AI’s Jev introduced a new category of ‘System One’ decision models that make fast, typed, probabilistic decisions from unstructured input, but it is a hosted API without self-hosting options. The vLLM community adapted DiffusionGemma, an open block-diffusion model already validated on Red Hat AI, to serve as a self-hostable decision engine using a fixed-length canvas and single denoising step.

## Approach / What changed

The vLLM community implemented a structured-read mode (PR #57250) that seeds a fixed-length canvas with answer templates, runs a single denoising step, and reads calibrated logprobs at each answer slot to return the top choice and confidence. Answers must be single tokens. If uncertainty is high, the client can re-read a few more times and measure agreement. This enables parallel decisions for multiple questions in one request.

## Takeaways

- Decision models like Jev provide guaranteed structure, default probabilities, and low latency for high-volume decisions, but are hosted APIs without self-hosting options.
- The vLLM community adapted DiffusionGemma to serve as a self-hostable decision engine using a fixed-length canvas and single denoising step, achieving over 300 decisions per second.
- DiffusionGemma is already validated on Red Hat AI, and the structured-read support in vLLM nightly allows teams to run decision models on their own infrastructure, including air-gapped environments.

## FAQ

### How does the decision engine work?

The engine uses a fixed-length canvas seeded with answer templates, runs a single denoising step, and reads calibrated logprobs at each answer slot to return the top choice and confidence. If uncertainty is high, the client can re-read a few more times and measure agreement.

### What are the performance numbers?

On a single DGX Spark, it achieved 8.7 req/s at 0.12 s for single requests, and 54 req/s at 0.58 s for 32 concurrent requests, answering three questions per request. Google's Cloud Run deployment reported 35–60 ms latency and 100–123 req/s at batch 32, translating to over 300 decisions per second.

### How can I run decision models on Red Hat AI?

DiffusionGemma is already validated on Red Hat AI. You can use the vLLM nightly with structured-read support on Red Hat AI Inference preview builds for prototyping. It will be supported in a future stable release after vLLM 0.31.0 is picked up.

**Tags:** [Decision Models](https://yomu.fyi/topic/decision-models), [DiffusionGemma](https://yomu.fyi/topic/diffusiongemma), [Red Hat AI](https://yomu.fyi/topic/red-hat-ai), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/28/run-decision-model-vllm-and-red-hat-ai)
- Source URL: https://developers.redhat.com/articles/2026/09/28/run-decision-model-vllm-and-red-hat-ai
- Ingested by Yomu: 2026-09-28T15:00:39.148Z

[Read original post](https://developers.redhat.com/articles/2026/09/28/run-decision-model-vllm-and-red-hat-ai)