Hugging Face Reads, Feb. 2021 - Long-range Transformers

Summary

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.

Context

Standard transformer self-attention incurs a quadratic computational and memory cost with respect to sequence length, creating a major bottleneck when scaling models beyond traditional 512- or 1024-token contexts in natural language processing and other domains like speech or protein modeling.

Approach / What changed

Researchers developed efficient transformer variants utilizing custom attention patterns, memory recurrence, low-rank projections, and kernel approximations. Specifically, architectures like Longformer combine local windowed and task-specific global attention, Compressive Transformers add compressed past activations, Linformer uses low-rank decomposition, and Performer approximates softmax attention kernels directly.

Takeaways

• Longformer uses dilated windowed attention and selective global tokens to scale memory linearly, serving as a drop-in replacement for standard self-attention layers in pre-trained models.
• Relative positional embeddings generalize effectively to unseen sequence lengths but remain computationally expensive, whereas absolute embeddings are cheaper but rigid.
• Staged pre-training that begins with short sequences and incrementally increases context length speeds up training and reduces reliance on spurious correlations.

Related reading

• huggingface.co · Sep 24, 2021

  Summer at Hugging Face

  Hugging Face released several major updates across its ecosystem, expanding its public Hub repository to over 16,000 models. Platform additions include Spaces for deploying Gradio and Streamlit demo applications, automatic TensorBoard instances, and evaluation metric tracking integrated with Papers With Code leaderboards. The Transformers library gained JAX/Flax support across more than 5,000 models, improved TensorFlow implementations, and introduced the transformers.onnx export module for model conversion. In research, the BigScience project completed large-scale training of a 13-billion-parameter English decoder model on Jean Zay, while the DeDLOC method enabled training the sahajBERT Bengali model without HPC infrastructure. Accepted conference papers detailed the Datasets library, prompt data-point equivalence, prompt-based fine-tuning heuristics, and block pruning techniques that reduced BERT size by 74 percent.

  system

• huggingface.co · Jul 3, 2020

  The Reformer - Pushing the limits of language modeling

  Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.