Loading…
A Practical Guide to GPU-Initiated Communication for Molecular Dynamics at Scale
NVIDIA Developer BlogMichelle Horton
Summary
Molecular dynamics packages like GROMACS face severe scaling bottlenecks when CPU-orchestrated MPI communication introduces blocking synchronizations during halo exchanges. To resolve this, researchers implemented GPU-initiated communication using NVIDIA NVSHMEM to eliminate CPU-GPU handoffs. The redesigned approach fuses packing, remote memory access, and completion waiting into device-side kernels using fine-grained signaling. Benchmarks on NVIDIA DGX H100 systems demonstrated up to 1.5x performance gains intra-node and up to 2x on GB200 NVL72 multi-node clusters, particularly on latency-bound systems. While device-initiated communication introduces a slight 1–3% overhead on very large compute-bound configurations at low node counts, it consistently improves strong scaling for production workloads.
Context
In GROMACS, CPU-orchestrated MPI communication creates blocking synchronization points across spatial dimensions during halo exchanges, consuming over 50% of CPU wall time at peak iteration rates and capping strong scaling.
Approach / What changed
Replaced CPU-centric MPI with GPU-native communication via NVSHMEM, fusing pack, put, and completion-wait operations into device-side kernels with fine-grained per-pulse signaling and interconnect-aware transports.
Takeaways
- Standard 3D domain decomposition halo exchanges in GROMACS require 12 blocking CPU-GPU synchronization waits per time-step when using CPU-orchestrated MPI.
- The NVSHMEM implementation replaces coarse phase-level barriers with fused pack-put-wait GPU kernels and GPU-visible signals to eliminate host synchronization.
- Intra-node benchmarks on DGX H100 showed up to a 1.5x speedup over GPU-aware MPI, reaching a 46% gain (1,649 vs. 1,126 ns/day) on a 45,000-atom system using four GPUs.
Related reading
Accelerating End-to-End Co-Folding Performance with NVIDIA BioNeMo Agent Toolkit
Biomolecular structure prediction and co-folding pipelines encounter latency and memory ceilings during sequence alignment and multi-residue inference. NVIDIA addresses these constraints through the BioNeMo Agent Toolkit, which connects AI agents to accelerated biological computing components. The pipeline offloads multiple sequence alignment to MMseqs2-GPU, while cuEquivariance and the OpenFold3 NIM accelerate core geometric attention kernels to cut inference latency up to fourfold. To handle large molecular assemblies beyond single-GPU memory limits, the Fold-CP framework distributes context across multiple GPUs with O(N²/P) memory scaling. Together, these tools enable the prediction of complexes up to 32,000 tokens on 64 NVIDIA B300 GPUs, making large-scale virtual screening and ribosome-scale modeling computationally feasible.
Elizabeth Goodman