Speeding up LLM inference with P-EAGLE in vLLM Speculators

Summary

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.

Takeaways

1. 1

   P-EAGLE verifies multiple drafted tokens in one target-model forward pass and accepts the correct prefix, preserving the verifier model’s output distribution.

2. 2

   COD sampling uses geometric position retention of 1.0, 0.7, 0.49 and so on, with a 0.2 floor, to control training memory for deeper prediction levels.

3. 3

   For the Qwen3-8B example, hidden states from layers 2, 18 and 33 are concatenated into a 3 × 4096 tensor, and training takes about 50 minutes on four H100s with 5,000 samples.