# Hugging Face Reads, Feb. 2021 - Long-range Transformers

huggingface.co · Victor Sanh · Mar 9, 2021

**Type:** Explainer

## Summary

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.

## Context

Standard transformer self-attention incurs a quadratic computational and memory cost with respect to sequence length, creating a major bottleneck when scaling models beyond traditional 512- or 1024-token contexts in natural language processing and other domains like speech or protein modeling.

## Approach / What changed

Researchers developed efficient transformer variants utilizing custom attention patterns, memory recurrence, low-rank projections, and kernel approximations. Specifically, architectures like Longformer combine local windowed and task-specific global attention, Compressive Transformers add compressed past activations, Linformer uses low-rank decomposition, and Performer approximates softmax attention kernels directly.

## Takeaways

- Longformer uses dilated windowed attention and selective global tokens to scale memory linearly, serving as a drop-in replacement for standard self-attention layers in pre-trained models.
- Relative positional embeddings generalize effectively to unseen sequence lengths but remain computationally expensive, whereas absolute embeddings are cheaper but rigid.
- Staged pre-training that begins with short sequences and incrementally increases context length speeds up training and reduces reliance on spurious correlations.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://huggingface.co/blog/long-range-transformers)