Perceiver IO: a scalable, fully-attentional model that works on any modality

Summary

Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.

Takeaways

1. 1

   Perceiver IO scales linearly with input size during cross-attention, while its self-attention layer compute depends entirely on the fixed number of latents.

2. 2

   Because the architecture handles raw input dimensions efficiently, text models can process raw UTF-8 byte sequences without subword tokenizers like BPE or WordPiece.

3. 3

   On ModelNet40 point cloud classification, Perceiver achieved 85.7% top-1 accuracy, and matching performance when replacing the Transformer in AlphaStar with an 87% win rate against the Elite bot.