---
title: "Hugging Face"
description: "AI platform and community providing machine learning models, datasets, libraries, and tools for building AI applications."
---

# Hugging Face
> AI platform and community providing machine learning models, datasets, libraries, and tools for building AI applications.

## Articles

### [Transformer-based Encoder-Decoder Models](https://yomu.fyi/post/transformer-based-encoder-decoder-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Oct 10, 2020
- Type: Explainer

Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.


### [Block Sparse Matrices for Smaller and Faster Language Models](https://yomu.fyi/post/block-sparse-matrices-for-smaller-and-faster-language-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: François Lagunas
- Published: Sep 10, 2020
- Type: Announcement

Full dense layers in neural networks are often unnecessarily large and can be pruned without sacrificing precision, but available sparse algebra tools and PyTorch native implementations lack efficiency. To resolve this issue, the pytorch\_block\_sparse extension introduces BlockSparseLinear as a drop-in replacement for standard linear layers alongside BlockSparseModelPatcher for on-the-fly model modifications. The library uses NVIDIA CUTLASS C++ CUDA templates derived from Yulhwa Kim's cutlass tilesparse proof of concept. Although baseline sparse operations remain roughly twice as slow as cuBLAS-optimized dense counterparts, performance scales with sparsity, making a 75% sparse matrix approximately two times faster than a dense layer while reducing memory consumption by four times. Future updates will target dynamic sparsity optimization and NVIDIA Ampere Tensor Core primitives.


### [The Reformer - Pushing the limits of language modeling](https://yomu.fyi/post/the-reformer-pushing-the-limits-of-language-modeling.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Jul 3, 2020
- Type: Explainer

Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.


### [How to generate text: using different decoding methods for language generation with Transformers](https://yomu.fyi/post/how-to-generate-text-using-different-decoding-methods-for-language-gen.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Mar 1, 2020
- Type: Tutorial

Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.


### [How to train a new language model from scratch using Transformers and Tokenizers](https://yomu.fyi/post/how-to-train-a-new-language-model-from-scratch-using-transformers-and.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Chaumond
- Published: Feb 14, 2020
- Type: Tutorial

Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.


[Newer posts](https://yomu.fyi/company/hugging-face/page/16.md)