--- title: "Hugging Face" description: "AI platform and community providing machine learning models, datasets, libraries, and tools for building AI applications." --- # Hugging Face > AI platform and community providing machine learning models, datasets, libraries, and tools for building AI applications. ## Articles ### [Training CodeParrot šŸ¦œ from Scratch](https://yomu.fyi/post/training-codeparrot-from-scratch.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Leandro von Werra - Published: Dec 8, 2021 - Type: Tutorial Training code generation models from scratch requires careful data filtering, tokenization, and compute management. To build CodeParrot, a 1.5-billion-parameter Python generation model, engineers extracted 20 million Python files from BigQuery and cleaned the resulting 180 GB corpus down to 50 GB after discovering extreme duplication. A custom GPT-2 tokenizer was trained over streamed samples, and the GPT-2 large architecture was initialized with layer-scaled, full-precision attention mechanisms. The training pipeline leveraged Hugging Face Accelerate alongside gradient checkpointing and a streaming iterable dataset designed to yield fixed-length concatenated token sequences. CodeParrot was trained on roughly 25 to 30 billion tokens and evaluated on coding tasks using the pass@k metric. ### [Introducing Snowball Fight ā˜ƒļø, our first ML-Agents environment](https://yomu.fyi/post/introducing-snowball-fight-our-first-ml-agents-environment.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Thomas Simonini - Published: Dec 2, 2021 - Type: Announcement Hugging Face has released Snowball Fight 1vs1, its first custom Deep Reinforcement Learning environment created using the Unity Machine Learning Agents Toolkit. The game enables players to shoot snowballs against a trained agent and is hosted on Hugging Face Spaces. This release marks the beginning of an effort to build an ecosystem for Deep Reinforcement Learning researchers and enthusiasts using Unity ML-Agents. The initiative focuses on building open-source environments, hosting models and training environments on the Hugging Face Hub, and deploying interactive demos via Spaces. Future plans include publishing technical tutorials, releasing additional environments, and developing a Snowball Fight 2vs2 version that utilizes the MA-POCA algorithm to train cooperative team behaviors. ### [Getting Started with Hugging Face Transformers for IPUs with Optimum](https://yomu.fyi/post/getting-started-with-hugging-face-transformers-for-ipus-with-optimum.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Tim Santos, Julien Simon - Published: Nov 30, 2021 - Type: Tutorial Transformer prediction speed can hinder latency-sensitive workloads like conversational systems and search, while optimizing hardware performance requires significant effort. Hugging Face introduced Optimum, an open-source library that reduces Transformer model latency across diverse hardware targets. Through a partnership with Graphcore, BERT is introduced as the initial IPU-optimized model for parallel AI hardware. Developers configure the Poplar SDK environment and PopTorch inside Graphcloud, install optimum-graphcore, and execute the run\_qa.py script with an IPU configuration file. Fine-tuning bert-base-uncased on SQuAD v1.1 using the IPUTrainer class yields an evaluation F1 score of 88.2757 and an exact match score of 80.6623 across 10,784 evaluation samples. ### [Introducing the Data Measurements Tool: an Interactive Tool for Looking at Datasets](https://yomu.fyi/post/introducing-the-data-measurements-tool-an-interactive-tool-for-looking.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Sasha Luccioni, Yacine Jernite, Margaret Mitchell - Published: Nov 29, 2021 - Type: Announcement Hugging Face released an open-source Python library and no-code interface named the Data Measurements Tool to calculate metrics for responsible dataset curation. In AI development, scraped data often enters training pipelines without detailed analysis of representation or downstream model influence, partly because querying raw datasets typically requires complex coding. Built using Hugging Face Dataset and Spaces Hubs alongside Streamlit, the tool provides descriptive, distributional, and comparison statistics. It computes missing values, vocabulary sizes, duplicate counts, label distributions, and Zipf's law adherence to identify unnatural language patterns or markup artifacts. Additionally, it offers embedding clusters via a Sentence-Transformer model and normalized pointwise mutual information metrics across gender and sexual orientation identity terms. ### [Accelerating PyTorch distributed fine-tuning with Intel technologies](https://yomu.fyi/post/accelerating-pytorch-distributed-fine-tuning-with-intel-technologies.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Julien Simon - Published: Nov 19, 2021 - Type: Tutorial Transfer learning makes CPU-based clusters a viable, cost-effective option for shorter deep learning fine-tuning jobs. To accelerate PyTorch workloads, clusters can leverage Intel Ice Lake architecture featuring AVX-512 and Vector Neural Network Instructions alongside specialized software optimizations. Implementing the Intel Extension for PyTorch and the oneAPI Collective Communications Library (oneCCL) addresses communication bottlenecks across distributed nodes. In demonstrated tests fine-tuning transformer models on GLUE tasks across Amazon EC2 c6i.16xlarge instances, two-node jobs delivered roughly a 1.7x speedup, while four-node setups achieved a 3x speedup. These performance libraries and multi-node configurations maintain consistent scaling across both smaller datasets like MRPC and larger benchmarks like QQP. ### [Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with šŸ¤— Transformers](https://yomu.fyi/post/fine-tune-xlsr-wav2vec2-for-low-resource-asr-with-transformers.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Patrick von Platen - Published: Nov 15, 2021 - Type: Tutorial Fine-tuning the multilingual XLS-R speech model adapts pretrained cross-lingual audio representations to automatic speech recognition tasks with limited labeled data. The demonstration configures a Wav2Vec2-XLS-R-300M checkpoint using Connectionist Temporal Classification on roughly four hours of validated Turkish audio from Common Voice. Building the pipeline requires pairing a Wav2Vec2FeatureExtractor for audio signals with a custom Wav2Vec2CTCTokenizer derived from dataset transcriptions. Training with Hugging Face Trainer over 3,200 steps decreases the validation word error rate from 0.7000 down to 0.3195. While the resulting transcription demonstrates recognizable phonetic alignment, output quality can be further improved by extending training schedules, refining preprocessing, and adding language model decoding. ### [Scaling up BERT-like model Inference on modern CPU - Part 2](https://yomu.fyi/post/scaling-up-bert-like-model-inference-on-modern-cpu-part-2.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Ella Charlaix, Jeff Boudier, Morgan Funtowicz, Michael Benayoun - Published: Nov 4, 2021 - Type: Problem & solution Scaling transformer inference on modern CPUs requires coordinating hardware capabilities with software optimizations across memory allocation, parallel execution, and mathematical kernel libraries. Intel Ice Lake Xeon processors deliver up to seventy-five percent faster inference across natural language processing tasks compared to Cascade Lake by combining Sunny Cove architecture improvements with dedicated extensions like Intel Extension for PyTorch. Fine-tuning software knobs such as alternative allocators, threading runtimes like Intel OpenMP, and core allocations can involve thousands of possible configuration combinations. Rather than evaluating these combinations through exhaustive brute force, Bayesian optimization via Intel SigOpt achieves comparable latency within roughly twenty trials and reveals experiment-dependent parameter importance. These software tuning strategies enable low-latency production transformer serving without dedicated accelerators. ### [Course Launch Community Event](https://yomu.fyi/post/course-launch-community-event.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Sylvain Gugger - Published: Oct 26, 2021 - Type: Announcement Hugging Face scheduled the release of Part 2 of its machine learning course for November 15th alongside a collaborative community event. While Part 1 covered text classification and Model Hub uploads, Part 2 expands into token classification, causal and masked language modeling, translation, summarization, question answering, and ecosystem tools like Datasets and Tokenizers. The event includes two days of technical talks from industry practitioners covering transfer learning, ethics in machine learning development, and workflows across Keras, PyTorch, and TensorFlow. Participants collaborate on team projects to fine-tune natural language processing models and develop interactive portfolio demos, with AWS supplying complimentary compute through Amazon SageMaker. Attendees who successfully complete and demonstrate a project receive a certificate of completion. ### [Large Language Models: A New Moore's Law?](https://yomu.fyi/post/large-language-models-a-new-moore-s-law.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Julien Simon - Published: Oct 26, 2021 - Type: Problem & solution Recent advancements in generative artificial intelligence have spurred the rapid growth of large language models like Megatron-Turing NLG 530B, which require vast financial investments and substantial energy consumption. Training these massive parameter architectures demands hundreds of multi-GPU servers and generates significant carbon footprints for relatively modest benchmark gains. Rather than relying on brute-force scaling, engineering teams can adopt smaller, more frugal architectures through knowledge distillation and transfer learning techniques. Practical alternatives include fine-tuning existing pretrained models, deploying on energy-efficient cloud infrastructure, and applying optimizations like pruning, layer fusion, and quantization. These methods deliver low-latency inference and high task accuracy while significantly decreasing hardware requirements, development time, and environmental impact. ### [Train a Sentence Embedding Model with 1B Training Pairs](https://yomu.fyi/post/train-a-sentence-embedding-model-with-1b-training-pairs.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Antoine SIMOULIN - Published: Oct 25, 2021 - Type: Announcement Sentence embeddings map sentences into dense vectors to capture semantic meaning for downstream tasks such as clustering, asymmetric question answering, and text search. To train general-purpose models at scale, the team utilized a dataset compiled from multiple sources containing up to one billion sentence pairs. Training leveraged contrastive learning with Multiple Negative Ranking Loss and in-batch negatives, running across seven TPU v3-8 accelerators with JAX and Flax. Batches combined samples from at least two datasets to balance hard negatives within topics against broader cross-topic semantic structure. The project resulted in twenty trained modelsā€”including variants of RoBERTa, DistilBERT, and MPNetā€”alongside eight specialized evaluation datasets and an interactive demo covering sentence similarity, search, and gender bias. ### [The Age of Machine Learning As Code HasĀ Arrived](https://yomu.fyi/post/the-age-of-machine-learning-as-code-has-arrived.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Julien Simon - Published: Oct 20, 2021 - Type: Explainer Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction. ### [Fine tuning CLIP with Remote Sensing (Satellite) images and captions](https://yomu.fyi/post/fine-tuning-clip-with-remote-sensing-satellite-images-and-captions.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Arto, Dev Vidhani, Goutham, Mayank Bhaskar, Ritobrata Ghosh, Sujit Pal - Published: Oct 13, 2021 - Type: Problem & solution Pre-trained vision-language models like CLIP generalize well across everyday imagery but struggle with remote sensing domain characteristics. To enable text-based search across satellite imagery, a distributed team fine-tuned the openai/clip-vit-base-patch32 model on TPU hardware using JAX and Flax during the Hugging Face Flax/JAX Community Week. The training pipeline incorporated satellite image and caption pairs from the RSICD, UCM, and Sydney datasets alongside extensive image transforms and backtranslation-based text augmentation. Contrastive optimization using Adam with a learning rate of 5e-6 achieved top-1 retrieval accuracy of 0.883 compared to 0.572 on the baseline model. An interactive demo uses a NMSLib index for text-to-image matching, image-to-image matching, and patch-level feature detection. ### [Hosting your Models and Datasets on Hugging Face Spaces using Streamlit](https://yomu.fyi/post/hosting-your-models-and-datasets-on-hugging-face-spaces-using-streamli.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: merve - Published: Oct 5, 2021 - Type: Tutorial Streamlit enables developers to create interactive web demonstrations and visualizations for machine learning models and datasets. In an illustrative text completion interface recreating Write with Transformer, UI components such as text areas, sidebars, number inputs, and continuous sliders collect user text and inference hyperparameters. For data inspection, the streaming feature in Hugging Face Datasets loads large corpora directly into memory without requiring complete local downloads, integrating with pandas, matplotlib, and seaborn for tabular and chart rendering. Deploying these interactive applications to Hugging Face Spaces involves uploading the code files while defining dependencies inside a requirements.txt file. Matching local Streamlit versions ensures consistent runtime execution when hosting models and datasets online. ### [Showcase Your Projects in Spaces using Gradio](https://yomu.fyi/post/showcase-your-projects-in-spaces-using-gradio.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: merve - Published: Oct 5, 2021 - Type: Tutorial Demonstrating machine learning projects to the community requires accessible hosting and minimal interface boilerplate. Gradio integrates directly with the Hugging Face Hub and its underlying Inference API, enabling developers to load models using repository identifiers and launch interactive interfaces with a few lines of code. Hugging Face Spaces provides a free hosting environment where users select Gradio as the SDK, commit an app.py script, and deploy live web demonstrations for frameworks like Transformers, spaCy, SpeechBrain, and Asteroid. For complex workflows, developers can chain multiple models sequentially via Gradio Series or host custom checkpoints directly when models lack native Inference API support. ### [Summer at Hugging Face](https://yomu.fyi/post/summer-at-hugging-face.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: system - Published: Sep 24, 2021 - Type: Announcement Hugging Face released several major updates across its ecosystem, expanding its public Hub repository to over 16,000 models. Platform additions include Spaces for deploying Gradio and Streamlit demo applications, automatic TensorBoard instances, and evaluation metric tracking integrated with Papers With Code leaderboards. The Transformers library gained JAX/Flax support across more than 5,000 models, improved TensorFlow implementations, and introduced the transformers.onnx export module for model conversion. In research, the BigScience project completed large-scale training of a 13-billion-parameter English decoder model on Jean Zay, while the DeDLOC method enabled training the sahajBERT Bengali model without HPC infrastructure. Accepted conference papers detailed the Datasets library, prompt data-point equivalence, prompt-based fine-tuning heuristics, and block pruning techniques that reduced BERT size by 74 percent. ### [Hugging Face and Graphcore partner for IPU-optimized Transformers](https://yomu.fyi/post/hugging-face-and-graphcore-partner-for-ipu-optimized-transformers.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Sally Doherty - Published: Sep 14, 2021 - Type: Announcement Hugging Face launched its Hardware Partner Program at the 2021 AI Hardware Summit to offer device-optimized models and software integrations. As a founding member of the program, Graphcore collaborated with Hugging Face to optimize Transformer models for its Intelligence Processing Unit (IPU). Graphcore IPUs use a massively parallel MIMD architecture with on-die ultra-high bandwidth memory designed for AI workloads featuring fine-grained parallelism, low precision arithmetic, and sparsity. Through Hugging Face's open-source Optimum toolkit, developers will receive plug-and-play access to certified, hardware-optimized models across vision, speech, translation, and text generation. Graphcore's Poplar SDK integrates with PyTorch, TensorFlow, Docker, and Kubernetes, simplifying the process of porting workloads to IPU-POD datacenter compute systems. ### [Introducing Optimum: The Optimization Toolkit for Transformers at Scale](https://yomu.fyi/post/introducing-optimum-the-optimization-toolkit-for-transformers-at-scale.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Morgan Funtowicz, Ella Charlaix, Michael Benayoun, Jeff Boudier - Published: Sep 14, 2021 - Type: Announcement Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads. ### [Deep Learning over the Internet: Training Language Models Collaboratively](https://yomu.fyi/post/deep-learning-over-the-internet-training-language-models-collaborative.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Max Ryabinin, Lucile Saulnier - Published: Jul 15, 2021 - Type: Problem & solution Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers. ### [Welcome spaCy to the Hugging Face Hub](https://yomu.fyi/post/welcome-spacy-to-the-hugging-face-hub.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Omar Sanseviero, Ines Montani - Published: Jul 13, 2021 - Type: Announcement Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines. ### [Deploy Hugging Face models easily with Amazon SageMaker](https://yomu.fyi/post/deploy-hugging-face-models-easily-with-amazon-sagemaker.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Philipp Schmid - Published: Jul 8, 2021 - Type: Announcement Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF\_MODEL\_ID and HF\_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity. [Newer posts](https://yomu.fyi/company/hugging-face/page/14.md) Ā· [Older posts](https://yomu.fyi/company/hugging-face/page/16.md)