Loading…
Hugging Face
AI platform and community providing machine learning models, datasets, libraries, and tools for building AI applications.
- 325 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Sentence Transformers in the Hugging Face Hub
Sentence Transformers has collaborated with the Hugging Face Hub as part of its v2 release to make sentence, paragraph, and image embedding models readily accessible. The Hub now hosts over 90 pretrained models supporting more than 100 languages for use cases like semantic search and multilingual zero-shot classification. Alongside direct model loading in minimal code, the platform provides interactive Hub widgets for feature extraction and sentence similarity computation. Developers can execute inference programmatically through Inference API endpoints. Additionally, the library supports direct model uploading to the Hub via save_to_hub, which automatically generates a model card detailing layer architecture and dual usage with Sentence Transformers and Transformers.
Few-shot learning in practice: GPT-Neo and the 🤗 Accelerated Inference API
Limited labeled data often prevents machine learning models from achieving high performance during traditional fine-tuning. Few-shot learning addresses this hurdle in natural language processing by supplying a short task description and a few demonstration examples at inference time. EleutherAI developed GPT-Neo, an open transformer-based model trained on the Pile dataset, which can execute few-shot tasks via the Hugging Face Accelerated Inference API. Compared to larger models like GPT-3, the 2.7B parameter version of GPT-Neo requires three to four examples alongside tuned hyperparameters like temperature and stopping sequences to generate accurate completions. Because few-shot learning relies heavily on pre-trained associations, careful output monitoring and human feedback mechanisms remain necessary to prevent automated bias.
Using & Mixing Hugging Face Models with Gradio 2.0
The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.
Scaling-up BERT Inference on CPU (Part 1)
Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.
Introducing 🤗 Accelerate
🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.
Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker
Hugging Face and Amazon SageMaker introduced optimized Deep Learning Containers alongside a SageMaker Python SDK estimator to streamline distributed transformer model training. This workflow demonstrates fine-tuning the 400-million-parameter facebook/bart-large-cnn model for dialogue summarization on the 16,000-conversation samsum dataset. Using SageMaker Data Parallelism enabled in the distribution configuration, training executes across two ml.p3dn.24xlarge compute instances equipped with sixteen total GPUs and a cumulative batch size of 64. The HuggingFace estimator pulls the run_summarization.py script directly from GitHub, enables mixed precision with fp16, and tracks evaluation performance through ROUGE scores. Following completion, the unzipped model artifacts and a generated model card are pushed to the Hugging Face Hub using the huggingface_hub library for hosted inference.
Understanding BigBird's Block Sparse Attention
Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.
The Partnership: Amazon SageMaker and Hugging Face
Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.
My Journey to a serverless transformers pipeline on Google Cloud
The post documents Maxence Dominici’s effort to deploy a Transformers sentiment-analysis pipeline on Google Cloud for classifying customer reviews from Discord. After considering AI Platform Prediction and App Engine, he chose Cloud Run with a Docker image because custom checkpoint handling, dependencies, and resource requirements made the other options unsuitable. The implementation uses a PyTorch DistilBERT model fine-tuned on SST-2, Flask, Gunicorn, and a GET request protected by an optional API key; the shown handler retrieves a review but passes the hard-coded text “that was great” to the classifier. Cloud Run is configured with 4 GB of memory, one worker, and one thread to limit memory pressure and billing. Requests take under five seconds including model loading, while cold starts may add about ten seconds, and the author reports that the pretrained model’s accuracy was already impressive.
Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers
Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.
Hugging Face Reads, Feb. 2021 - Long-range Transformers
Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.
Simple considerations for simple people building fancy neural networks
Building and training neural networks often presents frustrating debugging challenges because implementations can contain subtle errors while still running without crashes and achieving decent performance. The author shares a mental framework derived from natural language processing research to systematically construct and debug deep learning workflows. Practitioners should begin with thorough qualitative and quantitative dataset analysis before implementing simple baseline models to establish performance benchmarks. When coding the network, developers should verify implementation correctness by overfitting a tiny batch without regularization and tracking training dynamics such as losses, parameters, and gradients. Hyperparameter optimization should prioritize understanding underlying component behaviors through targeted random grid searches rather than launching hundreds of blind runs.
Retrieval Augmented Generation with Huggingface Transformers and Ray
Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.
Hugging Face on PyTorch / XLA TPUs
PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.
Faster TensorFlow models in Hugging Face Transformers
Hugging Face updated Transformers to improve the computational performance and robustness of its TensorFlow models, including BERT, RoBERTa, ELECTRA, and MPNet. In version 4.2.0, the optimized BERT implementation runs up to approximately 10 percent faster than Google's official implementation and twice as fast as Transformers release 4.1.1 when tested on a V100 GPU. The release also improves TensorFlow SavedModel generation by allowing flexible sequence lengths across runs, exposing all model inputs, and grouping hidden states or attention outputs into single tensors. In addition, users can customize serving signatures by subclassing models and decorating custom methods with explicit input specifications. Models exported to the SavedModel format can be packaged into Docker containers and served via TensorFlow Serving using either HTTP REST or gRPC APIs for production inference.
Fit More and Train Faster With ZeRO via DeepSpeed and FairScale
Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.
How we sped up transformer inference 100x for 🤗 API customers
Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models
Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.
Porting fairseq wmt19 translation system to transformers
The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.
Hyperparameter Search with Transformers and Ray Tune
Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.