Perceiver IO: a scalable, fully-attentional model that works on any modality

Summary

Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.

Context

Standard Transformer architectures scale poorly in compute and memory due to self-attention computing pairwise dot products across all inputs, forcing existing models to discretize or preprocess high-dimensional modalities like audio, images, and video into token sequences.

Approach / What changed

Perceiver IO applies cross-attention between arbitrary inputs and a compact set of latent variables (typically 256 or 512), executes self-attention only within this cheap latent space, and uses cross-attention again to produce arbitrary outputs. Hugging Face Transformers implements this via PerceiverModel with modular, optional preprocessors, decoders, and postprocessors.

Takeaways

• Perceiver IO scales linearly with input size during cross-attention, while its self-attention layer compute depends entirely on the fixed number of latents.
• Because the architecture handles raw input dimensions efficiently, text models can process raw UTF-8 byte sequences without subword tokenizers like BPE or WordPiece.
• On ModelNet40 point cloud classification, Perceiver achieved 85.7% top-1 accuracy, and matching performance when replacing the Transformer in AlphaStar with an 87% win rate against the Elite bot.

Related reading

• huggingface.co · Mar 2, 2022

  BERT 101 - State Of The Art NLP Model Explained

  Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions.

  Britney Muller

• huggingface.co · Oct 10, 2020

  Transformer-based Encoder-Decoder Models

  Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.