ModelExpress: Distributing Model Artifacts at the Speed of Light

Summary

Distributing large language model weights and JIT kernel caches across clusters imposes high latency on cold starts, autoscaling, and reinforcement learning loops. NVIDIA ModelExpress minimizes this delay by identifying existing compatible weight replicas across the cluster rather than treating every worker as an independent cold start. Initial replicas bootstrap by streaming directly from remote object storage into GPU memory or bypassing host staging using GPUDirect Storage. Subsequent replicas load pre-processed weights and kernel caches directly from serving peers over peer-to-peer RDMA via the NVIDIA Inference Xfer Library. In testing on DeepSeek-V4 Pro, this peer-to-peer artifact distribution transferred weights and caches in under ten seconds, reducing overall replica startup time from eight minutes to one minute and forty-four seconds.

Takeaways

1. 1

   MX transfers DeepSeek-V4 Pro weights and JIT kernel cache artifacts between replicas in under 10 seconds, cutting overall startup time from 8 minutes to 1 minute 44 seconds.

2. 2

   For initial bootstrap from object storage, Model Streamer pipelines concurrent tensor reads directly into GPU memory via bounded host staging without writing to local disk.

3. 3

   For RL post-training, MX uses a receiver-driven refit model where rollout workers discover published trainer tensors and pull directly via one-sided reads into their target layouts.