Hugging Face Reads, Feb. 2021 - Long-range Transformers

Summary

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.

Takeaways

1. 1

   Longformer uses dilated windowed attention and selective global tokens to scale memory linearly, serving as a drop-in replacement for standard self-attention layers in pre-trained models.

2. 2

   Relative positional embeddings generalize effectively to unseen sequence lengths but remain computationally expensive, whereas absolute embeddings are cheaper but rigid.

3. 3

   Staged pre-training that begins with short sequences and incrementally increases context length speeds up training and reduces reliance on spurious correlations.