Loading…
Supercharged Searching on the 🤗 Hub
Zachary Mueller
Summary
Programmatically searching the Hugging Face Hub previously required navigating web browser widgets or enduring trial and error to guess exact query string formats. The huggingface_hub library solves this issue by introducing helper utilities such as ModelSearchArguments, DatasetSearchArguments, and ModelFilter alongside HfApi. These namespace helpers translate accessible Python attributes into the formatted parameters expected by the backend API, covering datasets, tasks, and libraries. For complex queries across multiple tasks, frameworks, and datasets, developers can combine criteria inside ModelFilter instances and retrieve matching model metadata via api.list_models. Under the hood, the library uses AttributeDictionary, a data structure inspired by fastcore that enables nested tab-completion while supporting standard dictionary key indexing for special characters.
Context
Searching the Hugging Face Hub programmatically using the huggingface_hub library previously required trial and error to format search parameters correctly, or forced users to leave their IDE and Jupyter interfaces to search manually via web browser widgets.
Approach / What changed
The huggingface_hub library introduces ModelSearchArguments, DatasetSearchArguments, and ModelFilter helper classes to interface with HfApi. Built on AttributeDictionary (inspired by fastcore's AttrDict), these classes provide human-readable attributes with tab-completion that automatically format search parameters for API queries.
Takeaways
- ModelSearchArguments provides pre-formatted API search parameters as attributes (such as mapping model_args.dataset.glue to dataset:glue) to eliminate parameter syntax guesswork.
- ModelFilter allows combining multiple search constraints, including lists of task tags, training datasets, and supported frameworks, into a single structured query passed to HfApi.list_models.
- AttributeDictionary enables nested tab-completion and attribute deletion, but keys containing numbers or special characters must be accessed using standard dictionary bracket indexing instead of dot notation.
Related reading
huggingface.co ·
CO2 Emissions and the 🤗 Hub: Leading the Charge
Training and deploying machine learning models produces greenhouse gas emissions determined by infrastructure runtime, hardware type, and energy grid carbon intensity. Hugging Face introduced mechanisms to measure, report, and filter models based on their carbon footprint to improve ecological transparency. Using the huggingface_hub library, practitioners can query HfApi.list_models with an emissions_thresholds parameter to filter models by maximum or minimum carbon emissions measured in grams. In addition, installing the codecarbon package enables the Transformers Trainer to automatically attach CodeCarbonCallback and log training emissions to an emissions.csv file. Users can then publish these measured values directly into model card metadata under co2_eq_emissions for public tracking.
Sasha Luccioni, Zachary Mueller, Nate Rawhuggingface.co ·
Welcome Stable-baselines3 to the Hugging Face Hub 🤗
Hugging Face announced an official integration with Stable-Baselines3, a popular PyTorch library for training and testing Deep Reinforcement Learning agents across diverse environments like Gym, Atari, MuJoco, and Procgen. The integration enables researchers and developers to host their saved reinforcement learning checkpoints on the Hugging Face Hub and download pre-trained community models. Interacting with the Hub requires installing the huggingface_hub and huggingface_sb3 packages, which supply helper methods for authentication, downloading, and uploading. Practitioners can retrieve checkpoint zip files using the load_from_hub function by providing the target repository identifier and filename before loading them into Stable-Baselines3 algorithms. Furthermore, users authenticated via CLI or notebook login can train policies such as PPO and publish their saved zip files to the Hub using push_to_hub.