A Practical Guide to GPU-Initiated Communication for Molecular Dynamics at Scale

Summary

Molecular dynamics packages like GROMACS face severe scaling bottlenecks when CPU-orchestrated MPI communication introduces blocking synchronizations during halo exchanges. To resolve this, researchers implemented GPU-initiated communication using NVIDIA NVSHMEM to eliminate CPU-GPU handoffs. The redesigned approach fuses packing, remote memory access, and completion waiting into device-side kernels using fine-grained signaling. Benchmarks on NVIDIA DGX H100 systems demonstrated up to 1.5x performance gains intra-node and up to 2x on GB200 NVL72 multi-node clusters, particularly on latency-bound systems. While device-initiated communication introduces a slight 1–3% overhead on very large compute-bound configurations at low node counts, it consistently improves strong scaling for production workloads.

Takeaways

1. 1

   Standard 3D domain decomposition halo exchanges in GROMACS require 12 blocking CPU-GPU synchronization waits per time-step when using CPU-orchestrated MPI.

2. 2

   The NVSHMEM implementation replaces coarse phase-level barriers with fused pack-put-wait GPU kernels and GPU-visible signals to eliminate host synchronization.

3. 3

   Intra-node benchmarks on DGX H100 showed up to a 1.5x speedup over GPU-aware MPI, reaching a 46% gain (1,649 vs. 1,126 ns/day) on a 45,000-atom system using four GPUs.