Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo

Summary

Pinterest describes building a VLM serving stack on NVIDIA Blackwell B200 GPUs and NVIDIA Dynamo for multimodal products including Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, and content safety. Unlike text-only workloads, requests may contain thousands of images, costly and variable prefill, long multi-turn histories, and large KV caches, making preprocessing, routing, memory, and latency central serving concerns. The implementation combines Dynamo's distributed orchestration, multimodal frontend and vLLM support, KV-aware routing, disaggregated encoder/prefill/decode serving, and LMCache-based KV offloading with vLLM, an OpenAI Chat Completions interface, model-aware routing, EKS, and custom deployment components. Production-oriented AIPerf DAG benchmarks model QPS patterns, multi-turn conversations, many-image requests, and agentic DAGs; the team concludes that payload design, cache-aware routing, offload tiers, and a shared platform are essential to meeting product targets and accelerating new launches.

Takeaways

1. 1

   VLM traffic can include thousands of images per request, making image preprocessing and multimodal prompt construction significant serving concerns.

2. 2

   Dynamo's multimodal KV-aware routing, encoder/prefill/decode disaggregation, and LMCache-based KV offloading address prefill-heavy workloads and KV cache pressure.

3. 3

   AIPerf-based DAG benchmarks reproduce production QPS patterns, multi-turn conversations, many-image requests, and agentic DAGs rather than relying on single-shot synthetic prompts.