# Understanding BigBird's Block Sparse Attention

huggingface.co · Vasudev Gupta · Mar 31, 2021

**Type:** Explainer

## Summary

Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.

## Context

Standard transformer models exhibit O(n^2) time and memory complexity, which makes processing long sequences (n > 512) computationally expensive and creates severe memory constraints.

## Approach / What changed

BigBird replaces standard full attention with block sparse attention by combining sliding local tokens, global tokens (first and last blocks), and random tokens into block-based computations.

## Takeaways

- BigBird handles sequence lengths up to 4096 tokens by approximating full attention with block sparse attention.
- The sparse attention mechanism combines sliding window tokens, global tokens, and randomly selected tokens to capture local and long-range dependencies efficiently.
- In Hugging Face Transformers, the sequence length must satisfy specific lower bounds relative to block configurations, or the implementation falls back to original\_full attention.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/big-bird)