---
title: "Machine Learning"
description: "622 posts about Machine Learning, summarised, each linking to the original."
---

# Machine Learning
> 622 posts about Machine Learning, summarised, each linking to the original.

## Articles

### [Using & Mixing Hugging Face Models with Gradio 2.0](https://yomu.fyi/post/using-mixing-hugging-face-models-with-gradio-2-0.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Abubakar Abid
- Published: May 25, 2021
- Type: Announcement

The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.


### [Scaling-up BERT Inference on CPU (Part 1)](https://yomu.fyi/post/scaling-up-bert-inference-on-cpu-part-1.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Morgan Funtowicz
- Published: Apr 20, 2021
- Type: Benchmark

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.


### [How We Improved Agent Chat Efficiency with Machine Learning](https://yomu.fyi/post/how-we-improved-agent-chat-efficiency-with-machine-learning.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Suman Anand
- Published: Apr 19, 2021
- Type: Problem & solution

Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts.


### [Introducing 🤗 Accelerate](https://yomu.fyi/post/introducing-accelerate.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sylvain Gugger
- Published: Apr 16, 2021
- Type: Announcement

🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.


### [Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker](https://yomu.fyi/post/distributed-training-train-bart-t5-for-summarization-using-transformer.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Apr 8, 2021
- Type: Tutorial

Hugging Face and Amazon SageMaker introduced optimized Deep Learning Containers alongside a SageMaker Python SDK estimator to streamline distributed transformer model training. This workflow demonstrates fine-tuning the 400-million-parameter facebook/bart-large-cnn model for dialogue summarization on the 16,000-conversation samsum dataset. Using SageMaker Data Parallelism enabled in the distribution configuration, training executes across two ml.p3dn.24xlarge compute instances equipped with sixteen total GPUs and a cumulative batch size of 64. The HuggingFace estimator pulls the run\_summarization.py script directly from GitHub, enables mixed precision with fp16, and tracks evaluation performance through ROUGE scores. Following completion, the unzipped model artifacts and a generated model card are pushed to the Hugging Face Hub using the huggingface\_hub library for hosted inference.


### [Understanding BigBird's Block Sparse Attention](https://yomu.fyi/post/understanding-bigbird-s-block-sparse-attention.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Vasudev Gupta
- Published: Mar 31, 2021
- Type: Explainer

Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.


### [The Partnership: Amazon SageMaker and Hugging Face](https://yomu.fyi/post/the-partnership-amazon-sagemaker-and-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Mar 23, 2021
- Type: Announcement

Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.


### [My Journey to a serverless transformers pipeline on Google Cloud](https://yomu.fyi/post/my-journey-to-a-serverless-transformers-pipeline-on-google-cloud.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Dominici
- Published: Mar 18, 2021
- Type: Tutorial

The post documents Maxence Dominici’s effort to deploy a Transformers sentiment-analysis pipeline on Google Cloud for classifying customer reviews from Discord. After considering AI Platform Prediction and App Engine, he chose Cloud Run with a Docker image because custom checkpoint handling, dependencies, and resource requirements made the other options unsuitable. The implementation uses a PyTorch DistilBERT model fine-tuned on SST-2, Flask, Gunicorn, and a GET request protected by an optional API key; the shown handler retrieves a review but passes the hard-coded text “that was great” to the classifier. Cloud Run is configured with 4 GB of memory, one worker, and one thread to limit memory pressure and billing. Requests take under five seconds including model loading, while cold starts may add about ten seconds, and the author reports that the pretrained model’s accuracy was already impressive.


### [Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers](https://yomu.fyi/post/fine-tune-wav2vec2-for-english-asr-in-hugging-face-with-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Mar 12, 2021
- Type: Tutorial

Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.


### [Hugging Face Reads, Feb. 2021 - Long-range Transformers](https://yomu.fyi/post/hugging-face-reads-feb-2021-long-range-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Mar 9, 2021
- Type: Explainer

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.


### [4 Tips for Shipping Data Products Fast](https://yomu.fyi/post/4-tips-for-shipping-data-products-fast.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 2, 2021
- Type: Explainer

Shipping data products under tight deadlines requires teams to stay focused while avoiding overengineering and building something users will not use. Shopify’s Data Science team recommends four practices: run a timeboxed design sprint with a small team and a designated Champion, prototype quickly—often with spreadsheets—and gather feedback throughout development. For first iterations, the post advises using simple approaches instead of machine learning when complexity would delay integration, polish, and user conversations; a sales forecasting prototype used linear regression with two input variables before a more complex model was built. User research should use specific questions and a diverse set of users, as feedback changed a confusing 0–100 scoring display to easier-to-interpret letter grades. Together, these practices provide a clearer path to launching useful products quickly.


### [Simple considerations for simple people building fancy neural networks](https://yomu.fyi/post/simple-considerations-for-simple-people-building-fancy-neural-networks.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Feb 25, 2021
- Type: Explainer

Building and training neural networks often presents frustrating debugging challenges because implementations can contain subtle errors while still running without crashes and achieving decent performance. The author shares a mental framework derived from natural language processing research to systematically construct and debug deep learning workflows. Practitioners should begin with thorough qualitative and quantitative dataset analysis before implementing simple baseline models to establish performance benchmarks. When coding the network, developers should verify implementation correctness by overfitting a tiny batch without regularization and tracking training dynamics such as losses, parameters, and gradients. Hyperparameter optimization should prioritize understanding underlying component behaviors through targeted random grid searches rather than launching hundreds of blind runs.


### [Retrieval Augmented Generation with Huggingface Transformers and Ray](https://yomu.fyi/post/retrieval-augmented-generation-with-huggingface-transformers-and-ray.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Feb 10, 2021
- Type: Problem & solution

Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.


### [Roboflow.com choose Supabase to power Paint.wtf leaderboard](https://yomu.fyi/post/roboflow-com-choose-supabase-to-power-paint-wtf-leaderboard.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rory Wilding
- Published: Feb 9, 2021
- Type: Problem & solution

Brad Dwyer and Erik Dunteman created Paint.wtf to experiment with OpenAI's CLIP model, allowing users to draw images evaluated against text prompts and scored on a leaderboard. While the team initially considered Firebase, the platform lacked built-in document counting functionality essential for their leaderboard architecture. To avoid implementing custom workarounds or exporting data to external systems, they selected Supabase to leverage PostgreSQL's native counting capabilities. After launching on Hacker News, Reddit, and Product Hunt, the platform experienced viral traffic that peaked at over two drawing submissions per second and surpassed one hundred thousand participating users within twenty-four hours. The PostgreSQL-backed infrastructure maintained reliable performance throughout the surge, delivering accurate and up-to-date score rankings under heavy load.


### [Hugging Face on PyTorch / XLA TPUs](https://yomu.fyi/post/hugging-face-on-pytorch-xla-tpus.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ds, Lysandre
- Published: Feb 9, 2021
- Type: Announcement

PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer\_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.


### [Faster TensorFlow models in Hugging Face Transformers](https://yomu.fyi/post/faster-tensorflow-models-in-hugging-face-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Plu
- Published: Jan 26, 2021
- Type: Tutorial

Hugging Face updated Transformers to improve the computational performance and robustness of its TensorFlow models, including BERT, RoBERTa, ELECTRA, and MPNet. In version 4.2.0, the optimized BERT implementation runs up to approximately 10 percent faster than Google's official implementation and twice as fast as Transformers release 4.1.1 when tested on a V100 GPU. The release also improves TensorFlow SavedModel generation by allowing flexible sequence lengths across runs, exposing all model inputs, and grouping hidden states or attention outputs into single tensors. In addition, users can customize serving signatures by subclassing models and decorating custom methods with explicit input specifications. Models exported to the SavedModel format can be packaged into Docker containers and served via TensorFlow Serving using either HTTP REST or gRPC APIs for production inference.


### [Fit More and Train Faster With ZeRO via DeepSpeed and FairScale](https://yomu.fyi/post/fit-more-and-train-faster-with-zero-via-deepspeed-and-fairscale.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Jan 19, 2021
- Type: Benchmark

Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.


### [How we sped up transformer inference 100x for 🤗 API customers](https://yomu.fyi/post/how-we-sped-up-transformer-inference-100x-for-api-customers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nicolas Patry
- Published: Jan 18, 2021
- Type: Problem & solution

Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.


### [Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models](https://yomu.fyi/post/leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Nov 9, 2020
- Type: Tutorial

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.


### [Porting fairseq wmt19 translation system to transformers](https://yomu.fyi/post/porting-fairseq-wmt19-translation-system-to-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Nov 3, 2020
- Type: Tutorial

The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.


[Newer posts](https://yomu.fyi/topic/machine-learning/page/29.md) · [Older posts](https://yomu.fyi/topic/machine-learning/page/31.md)