Introducing Storage Buckets on the Hugging Face Hub

Summary

Hugging Face introduces Storage Buckets as a mutable, S3-like object storage layer on the Hub for intermediate machine-learning artifacts such as checkpoints, optimizer states, processed shards, logs, and traces. Unlike versioned Models and Datasets repositories, Buckets support frequent writes, overwrites, directory synchronization, and cleanup while retaining standard Hub permissions and public or private visibility. Built on Xet, they chunk and deduplicate content across files, reducing repeated bytes, bandwidth, transfer time, and, for Enterprise customers, billed storage. The CLI and Python APIs support creation, syncing, dry runs, plans, inspection, individual file operations, and programmatic workflows; HfFileSystem and fsspec also allow compatible libraries to read and write through hf:// paths. Pre-warming can place hot data near AWS or GCP compute, while roadmap work aims to connect Buckets directly with versioned repos.

Takeaways

1. 1

   Buckets are mutable, non-versioned Hub storage containers that can be public or private and addressed with hf://buckets paths.

2. 2

   Xet deduplicates chunks across related artifacts such as processed datasets and successive checkpoints, improving transfer efficiency and reducing Enterprise billed storage.

3. 3

   The hf CLI, huggingface_hub, JavaScript client, and HfFileSystem support bucket management and integration with programmatic and fsspec-compatible workflows.