Streaming datasets: 100x More Efficient

Summary

Training machine learning models on multi-terabyte datasets often requires long download times or suffers from network request storms when multiple DataLoader workers initialize independently. Hugging Face revamped the backend of its datasets library to make dataset streaming viable for large-scale distributed training without modifying the standard API. Startup latency was reduced by caching resolved data file lists across workers and bundling API calls, cutting initial request volume up to one hundred times. The team also implemented background prefetching for Parquet files and added configurable buffering parameters to keep GPUs fully saturated during training runs. Benchmark tests demonstrated up to twice the sample processing throughput and zero worker crashes across 256 concurrent workers on 64 H100 GPUs.

Context

Multi-terabyte machine learning training runs often required hours to pre-download datasets or caused request rate limiting on the Hugging Face Hub. For example, testing nanoVLM generated over 100,000 requests in under a minute because every DataLoader worker independently initialized the dataset and resolved file lists, resulting in blocked IP addresses and high startup latency.

Approach / What changed

The Hugging Face datasets backend was optimized for startup and streaming phases while maintaining backward compatibility with load_dataset(..., streaming=True). Startup optimizations include caching file lists locally so only the first worker queries the Hub, alongside bundled resolution calls. Streaming optimizations include background prefetching for Parquet datasets, customizable PyArrow buffer sizes, integration with Xet deduplication via Parquet Content Defined Chunking, and HfFileSystem caching in torch DataLoaders.

Takeaways

• A persistent data files cache eliminates redundant Hub queries by allowing the initial DataLoader worker to resolve file lists and share them locally with other workers.
• Parquet prefetching and configurable PyArrow cache options, such as adjusting block size and prefetch limits, double streaming throughput and prevent GPU idle time.
• Hugging Face integrates Xet deduplication and Parquet Content Defined Chunking to speed up dataset uploads and remote streaming compared to traditional remote storage.

Related reading

• huggingface.co · Sep 24, 2021

  Summer at Hugging Face

  Hugging Face released several major updates across its ecosystem, expanding its public Hub repository to over 16,000 models. Platform additions include Spaces for deploying Gradio and Streamlit demo applications, automatic TensorBoard instances, and evaluation metric tracking integrated with Papers With Code leaderboards. The Transformers library gained JAX/Flax support across more than 5,000 models, improved TensorFlow implementations, and introduced the transformers.onnx export module for model conversion. In research, the BigScience project completed large-scale training of a 13-billion-parameter English decoder model on Jean Zay, while the DeDLOC method enabled training the sahajBERT Bengali model without HPC infrastructure. Accepted conference papers detailed the Datasets library, prompt data-point equivalence, prompt-based fine-tuning heuristics, and block pruning techniques that reduced BERT size by 74 percent.

  system

• Meta · Jul 1, 2026

  Meta’s AI Storage Blueprint at Scale

  Meta redesigned its BLOB-storage architecture on top of the foundational Tectonic block layer to eliminate GPU stalls and accelerate AI training workflows. The legacy storage system suffered from multi-layered metadata lookups, cross-region latency, and dataplane proxy bottlenecks that conflicted with the millisecond access requirements of flash-based AI clusters. To resolve these issues, the team collapsed disparate metadata into a single flat schema backed by ZippyDB for O(1) path lookups and introduced a fat client SDK capable of streaming data directly from storage servers. The updated stack also leverages spare GPU host memory as a distributed data cache alongside distributed read-plan caches to handle severe traffic spikes during checkpointing and model loading. These protocol and architectural changes deliver predictable latencies, reduce power consumption, and maintain high throughput during distributed training.