vLLM V0 to V1: Correctness Before Corrections in RL

Summary

PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed_logprobs alone did not resolve the training gap.

Takeaways

1. 1

   vLLM V1 returned raw-model logprobs by default, while PipelineRL expected logprobs from the processed distribution used by the sampler; setting logprobs-mode=processed_logprobs removed the mean offset but did not restore full training parity.

2. 2

   The parity configuration disabled prefix caching and async scheduling, while V1 weight updates used pause_generation(mode="keep", clear_cache=False), receive_weight_update, and resume_generation to approximate V0's behavior.

3. 3

   Matching the trainer's fp32 lm_head was required for final parity: the corrected V1 run tracked the V0 reward and metric trajectories, whereas processed_logprobs alone left the training mismatch.