Loading…
ModelExpress: Distributing Model Artifacts at the Speed of Light
NVIDIA Developer BlogElizabeth Goodman
Summary
Distributing large language model weights and JIT kernel caches across clusters imposes high latency on cold starts, autoscaling, and reinforcement learning loops. NVIDIA ModelExpress minimizes this delay by identifying existing compatible weight replicas across the cluster rather than treating every worker as an independent cold start. Initial replicas bootstrap by streaming directly from remote object storage into GPU memory or bypassing host staging using GPUDirect Storage. Subsequent replicas load pre-processed weights and kernel caches directly from serving peers over peer-to-peer RDMA via the NVIDIA Inference Xfer Library. In testing on DeepSeek-V4 Pro, this peer-to-peer artifact distribution transferred weights and caches in under ten seconds, reducing overall replica startup time from eight minutes to one minute and forty-four seconds.
Context
Moving model checkpoints spanning hundreds of gigabytes or terabytes creates significant latency across cold starts, rolling deployments, autoscaling replicas, and reinforcement learning rollout updates. Furthermore, JIT kernel compilation and autotuning add several minutes of warmup latency on fresh replicas.
Approach / What changed
NVIDIA ModelExpress dynamically routes artifact loading through the fastest available path. When bootstrapping, it pipelines reads directly from object storage to GPU memory or uses GPUDirect Storage, while collapsing duplicate cluster downloads. For subsequent workers, it bypasses storage entirely by using peer-to-peer RDMA via NIXL to transfer resident weights, JIT kernel caches, and RL step updates directly between peer GPUs.
Takeaways
- MX transfers DeepSeek-V4 Pro weights and JIT kernel cache artifacts between replicas in under 10 seconds, cutting overall startup time from 8 minutes to 1 minute 44 seconds.
- For initial bootstrap from object storage, Model Streamer pipelines concurrent tensor reads directly into GPU memory via bounded host staging without writing to local disk.
- For RL post-training, MX uses a receiver-driven refit model where rollout workers discover published trainer tensors and pull directly via one-sided reads into their target layouts.
Related reading
NVIDIA NVLink: The Scale-Up Network for AI Factories
Scale-up networking has emerged as a critical architectural component for AI factories supporting trillion-parameter models, mixture-of-experts architectures, and disaggregated serving. NVIDIA presents its sixth-generation NVLink scale-up fabric and NVLink 6 Switch to address latency-sensitive all-to-all GPU communication and offload collective operations. In simulation benchmarks on models like DeepSeek-R1, Qwen 235B, and a simulated 2T parameter model, NVLink achieved up to 2.3X higher decode throughput than off-the-shelf Ethernet in a 72-accelerator domain. The architecture integrates NVLink-C2C to supply 1.8 TB/s of coherent CPU-GPU bandwidth in the Vera Rubin NVL72 platform, as well as NVLink Fusion for integrating semi-custom silicon XPUs. Through co-designed hardware, SHARP in-network compute, and NCCL libraries, the platform optimizes token throughput, factory uptime, and deployment economics.
Elizabeth Goodman