# Streaming datasets: 100x More Efficient

[Hugging Face](https://yomu.fyi/company/hugging-face) · Andres Marafioti, Quentin Lhoest, ben burtenshaw, Pedro Cuenca, merve · Oct 27, 2025

**Type:** Problem & solution

## Summary

Training machine learning models on multi-terabyte datasets often requires long download times or suffers from network request storms when multiple DataLoader workers initialize independently. Hugging Face revamped the backend of its datasets library to make dataset streaming viable for large-scale distributed training without modifying the standard API. Startup latency was reduced by caching resolved data file lists across workers and bundling API calls, cutting initial request volume up to one hundred times. The team also implemented background prefetching for Parquet files and added configurable buffering parameters to keep GPUs fully saturated during training runs. Benchmark tests demonstrated up to twice the sample processing throughput and zero worker crashes across 256 concurrent workers on 64 H100 GPUs.

## Context

Multi-terabyte machine learning training runs often required hours to pre-download datasets or caused request rate limiting on the Hugging Face Hub. For example, testing nanoVLM generated over 100,000 requests in under a minute because every DataLoader worker independently initialized the dataset and resolved file lists, resulting in blocked IP addresses and high startup latency.

## Approach / What changed

The Hugging Face datasets backend was optimized for startup and streaming phases while maintaining backward compatibility with load\_dataset(..., streaming=True). Startup optimizations include caching file lists locally so only the first worker queries the Hub, alongside bundled resolution calls. Streaming optimizations include background prefetching for Parquet datasets, customizable PyArrow buffer sizes, integration with Xet deduplication via Parquet Content Defined Chunking, and HfFileSystem caching in torch DataLoaders.

## Takeaways

- A persistent data files cache eliminates redundant Hub queries by allowing the initial DataLoader worker to resolve file lists and share them locally with other workers.
- Parquet prefetching and configurable PyArrow cache options, such as adjusting block size and prefetch limits, double streaming throughput and prevent GPU idle time.
- Hugging Face integrates Xet deduplication and Parquet Content Defined Chunking to speed up dataset uploads and remote streaming compared to traditional remote storage.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://huggingface.co/blog/streaming-datasets)