When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving

Summary

Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.

Takeaways

1. 1

   On homogeneous clusters, colocated encoder workers avoid dedicating an entire GPU to the relatively lightweight vision encoder; heterogeneous tiers are more useful when lower-tier GPUs can host encoder work.

2. 2

   EPD does not optimize the 43% of TTFT spent before the ViT starts; the post points to parallel media decoding, embedding cache, and multimodal KV routing for other pipeline stages.

3. 3

   With parallel media decoding enabled, an encoder-only benchmark reported mean latency falling from 281.3 to 207.0 ms and P99 latency from 752.1 to 581.7 ms at matched throughput.