NVIDIATanya LenzTutorial
How to Read Data into GPU Memory in Python with NVIDIA GPUDirect Storage
Summary
GPU applications face growing datasets that demand efficient data movement to keep accelerators supplied. NVIDIA GPUDirect Storage (GDS) enables a direct path between storage and GPU memory through the nvidia-fs kernel module, bypassing CPU staging. KvikIO provides Python interfaces over cuFile with defaults of a 16 KiB GDS threshold and 4 MiB task size, and its compat_mode setting determines whether cuFile is invoked. This tutorial demonstrates reading a 1 GiB file into a CuPy buffer on an H100 system with NVMe and verifying the path by setting CUFILE_STATS_LEVEL=3 and inspecting gds_stats output. With four cuFile worker threads and max_request_parallelism set to four, the default task size divides the read into 256 cuFile requests, while a 1 GiB task size results in four 256 MiB pieces processed through bounce buffers with slab caching.
Context
GPU applications are processing ever-growing datasets, and moving data efficiently from storage is a necessity for keeping accelerators supplied. NVIDIA Magnum IO GPUDirect Storage (GDS) addresses this by enabling a direct data path between supported storage and GPU memory, avoiding staging through CPU memory.
Approach / What changed
The tutorial guides users through installing NVIDIA KvikIO, inspecting KvikIO and cuFile settings, reading a file into a CuPy GPU buffer using direct I/O, and verifying the data path by enabling level-three cuFile statistics and using gds_stats. It also covers cuFile execution settings that affect request decomposition, such as worker threads, max_request_parallelism, and min_io_threshold_size_kb.
Takeaways
KvikIO uses a compatibility mode (AUTO, ON, OFF) and a GDS threshold to decide whether to send I/O requests to cuFile or use its POSIX path that stages through host memory.
cuFile independently determines the final transfer path based on system support; reaching cuFile does not guarantee the accelerated GDS path was used, which is indicated by nvfs counters in gds_stats output.
Default KvikIO settings (4 MiB task size, 16 KiB GDS threshold) divide a 1 GiB read into 256 cuFile requests, and cuFile execution settings like worker threads and min_io_threshold_size_kb affect how requests are split and scheduled.
FAQ
How does KvikIO decide whether to use cuFile?
KvikIO uses the compat_mode setting (AUTO, ON, OFF) and gds_threshold. In AUTO mode, it tries cuFile and falls back to POSIX if unavailable. Reads smaller than the GDS threshold go through the POSIX path directly.
How can I verify that the accelerated GDS path was used for a read?
Set CUFILE_STATS_LEVEL=3 and run the program. While the process remains active, use gds_stats to inspect per-GPU statistics; operations labeled nvfs indicate the accelerated path, while posix indicates CPU-staged transfers.
What happens when a GPU buffer is not registered for storage DMA?
cuFile reads into a registered GPU bounce buffer and copies the data to the destination. Since GDS 1.17, these bounce buffers can be configured as slabs, and the nvidia-fs path uses slabs no larger than 16 MiB.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.