# Perceiver IO: a scalable, fully-attentional model that works on any modality

huggingface.co · Niels Rogge · Dec 15, 2021

**Type:** Announcement

## Summary

Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.

## Context

Standard Transformer architectures scale poorly in compute and memory due to self-attention computing pairwise dot products across all inputs, forcing existing models to discretize or preprocess high-dimensional modalities like audio, images, and video into token sequences.

## Approach / What changed

Perceiver IO applies cross-attention between arbitrary inputs and a compact set of latent variables (typically 256 or 512), executes self-attention only within this cheap latent space, and uses cross-attention again to produce arbitrary outputs. Hugging Face Transformers implements this via PerceiverModel with modular, optional preprocessors, decoders, and postprocessors.

## Takeaways

- Perceiver IO scales linearly with input size during cross-attention, while its self-attention layer compute depends entirely on the fixed number of latents.
- Because the architecture handles raw input dimensions efficiently, text models can process raw UTF-8 byte sequences without subword tokenizers like BPE or WordPiece.
- On ModelNet40 point cloud classification, Perceiver achieved 85.7% top-1 accuracy, and matching performance when replacing the Transformer in AlphaStar with an 87% win rate against the Elite bot.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://huggingface.co/blog/perceiver)