huggingface_hub v1.0: Five Years of Building the Foundation of Open Machine Learning

Summary

The Hugging Face Hub team released huggingface_hub v1.0, marking the library's transition into a mature foundation for open machine learning after five years of development. The major update migrates the underlying network backend to httpx and replaces the legacy hf_transfer tool with the hf_xet binary package to enable chunk-level file deduplication across repositories. In addition, the release introduces a redesigned Typer-based CLI using the streamlined hf command alongside explicit authentication functions and expanded inference primitives. Legacy components, including the Git-based Repository class and the older InferenceApi interface, have been removed to establish a cleaner and more maintainable architecture. Most dependent machine learning libraries remain compatible across versions, though transformers requires v0.x for version 4 and v1.x for its upcoming version 5 release.

Context

The Hugging Face Hub initially relied on Git wrappers embedded inside the transformers library to share model checkpoints. As machine learning models, datasets, and spaces expanded across thousands of external tools, the ecosystem required dedicated infrastructure that eliminated heavy Git LFS dependencies, simplified authentication, and supported scalable transfers for large model artifacts.

Approach / What changed

Hugging Face upgraded huggingface_hub to v1.0 by switching the HTTP backend to httpx, replacing hf_transfer with the chunk-level hf_xet transfer package, and introducing a redesigned Typer-based hf CLI. The release removed deprecated interfaces like the Git-based Repository class and InferenceApi in favor of direct HTTP methods, explicit auth utilities, and InferenceClient.

Takeaways

• Version 1.0 switches the HTTP client backend to httpx while making HfHubHttpError inherit from both requests and httpx base error classes to preserve error-handling compatibility.
• File transfers now standardize on hf_xet, replacing hf_transfer and leveraging 64KB chunk-level deduplication instead of file-level Git LFS deduplication.
• Legacy interfaces such as the Git-based Repository class, HfFolder, and InferenceApi have been removed in favor of direct HTTP methods, explicit authentication helpers, and InferenceClient.

Related reading

• Hugging Face · Dec 18, 2025

  Tokenization in Transformers v5: Simpler, Clearer, and More Modular

  Transformers v5 overhauls its tokenization framework by separating tokenizer architecture from trained vocabularies. In contrast to v4's dual slow Python and fast Rust files, v5 consolidates each model tokenizer into a single file defaulting to the Rust-backed TokenizersBackend. The pipeline stages—normalizer, pre-tokenizer, model algorithm such as BPE or Unigram, post-processor, and decoder—are now directly exposed and configurable rather than buried in serialized files. Practitioners can instantiate blank tokenizer architectures and train custom vocabularies directly from iterators using native methods like train_new_from_iterator while retaining model-specific formatting rules. The wrapper layer continues to bridge raw tokenization and model requirements by managing chat templates, context limits, and special token insertion.

  Ita Zaporozhets, Aritra Roy Gosthipaty, Arthur Zucker, Sergio Paniego, merve, Pedro Cuenca

• huggingface.co · May 6, 2022

  Welcome fastai to the Hugging Face Hub

  Hugging Face Hub has integrated with the fastai deep learning library, allowing practitioners to upload and download models directly through the huggingface_hub Python client. Fastai Learner objects, which bundle model architectures, data loaders, and loss functions, can be published to a designated repository namespace using push_to_hub_fastai. Uploaded models automatically receive Git-based version control, Git LFS support for large files, and generated model cards. Practitioners can retrieve and instantiate published models locally for inference or transfer learning via from_pretrained_fastai. The integration also supports Blurr, an ecosystem library that combines fastai with Hugging Face Transformers for sequence classification and related text modeling tasks.