---
title: "Speeding up LLM inference with P-EAGLE in vLLM Speculators"
description: "P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets."
---

# Speeding up LLM inference with P-EAGLE in vLLM Speculators

[Red Hat](https://yomu.fyi/company/red-hat) · Helen Zhao, Megan Flynn, Dipika Sikka · Sep 3, 2026

**Type:** Explainer

## Summary

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.

## Context

LLM generation is latency-bound because models produce tokens sequentially, requiring a full forward pass for each token. EAGLE-3 reduces this cost with a small draft model, but its autoregressive drafting introduces dependencies between predictions and limits parallelism, particularly for long reasoning traces.

## Approach / What changed

P-EAGLE extends EAGLE-3 with prediction depths that draft multiple tokens in one forward pass. Speculators v0.6.0 supports the full workflow from data preparation and vLLM hidden-state extraction through training and deployment, using COD sampling, learnable mask parameters and a custom flex-attention mask.

## Takeaways

- P-EAGLE verifies multiple drafted tokens in one target-model forward pass and accepts the correct prefix, preserving the verifier model’s output distribution.
- COD sampling uses geometric position retention of 1.0, 0.7, 0.49 and so on, with a 0.2 floor, to control training memory for deeper prediction levels.
- For the Qwen3-8B example, hidden states from layers 2, 18 and 33 are concatenated into a 3 × 4096 tensor, and training takes about 50 minutes on four H100s with 5,000 samples.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/03/speeding-llm-inference-p-eagle-vllm-speculators)
- Source URL: https://developers.redhat.com/articles/2026/09/03/speeding-llm-inference-p-eagle-vllm-speculators
- Ingested by Yomu: 2026-09-03T20:00:54.306Z

[Read original post](https://developers.redhat.com/articles/2026/09/03/speeding-llm-inference-p-eagle-vllm-speculators)