---
title: "vLLM V0 to V1: Correctness Before Corrections in RL"
description: "PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed_logprobs alone did not resolve the training gap."
---

# vLLM V0 to V1: Correctness Before Corrections in RL

[Hugging Face](https://yomu.fyi/company/hugging-face) · Rafael Pardinas, Ehsan Kamalloo · May 6, 2026

**Type:** Problem & solution

## Summary

PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm\_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed\_logprobs alone did not resolve the training gap.

## Context

During the vLLM V0-to-V1 migration, PipelineRL observed that trainer-side logprobs and reward diverged from the known-good V0 reference early in GSPO training. The discrepancy affected policy ratios, KL, entropy, clip rate, and reward, raising separate questions about logprob semantics, inference runtime behavior, and whether the RL objective required correction.

## Approach / What changed

The migration first restored backend parity with the V0 reference. The fixes enabled processed\_logprobs, made prefix caching and async scheduling explicit and disabled, matched V0's inflight weight-update behavior with pause, receive\_weight\_update, and resume calls, and used an fp32 lm\_head for the final projection. Objective-side corrections were deferred until backend correctness could be evaluated independently.

## Takeaways

- vLLM V1 returned raw-model logprobs by default, while PipelineRL expected logprobs from the processed distribution used by the sampler; setting logprobs-mode=processed\_logprobs removed the mean offset but did not restore full training parity.
- The parity configuration disabled prefix caching and async scheduling, while V1 weight updates used pause\_generation(mode="keep", clear\_cache=False), receive\_weight\_update, and resume\_generation to approximate V0's behavior.
- Matching the trainer's fp32 lm\_head was required for final parity: the corrected V1 run tracked the V0 reward and metric trajectories, whereas processed\_logprobs alone left the training mismatch.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Migrations](https://yomu.fyi/topic/migration), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Hugging Face](https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections)
- Source URL: https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections
- Ingested by Yomu: 2026-08-29T01:48:44.743Z

[Read original post](https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections)