---
title: "Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers"
description: "Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB."
---

# Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tom Aarsen, Antoine Chaffin, Raphael Sourty · Aug 18, 2026

**Type:** Tutorial

## Summary

Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB.

## Context

Sentence Transformers previously handled dense and sparse models but not late interaction. The post presents MultiVectorEncoder in v6.0 as a way to use ColBERT-style text retrieval and ColPali-style visual document retrieval through the same library API, while retaining token-level matching information that single-vector compression can lose.

## Approach / What changed

The implementation keeps one projected vector per token, encodes documents independently, and scores queries against documents with the MaxSim operator. The guide demonstrates loading PyLate and Stanford-NLP ColBERT checkpoints, using colpali-engine models for page images, connecting models to search workflows, and controlling index cost with compression, token pooling, or retrieve-and-rerank.

## Takeaways

- MaxSim selects the highest similarity between each query token and any document token, then sums those values; with L2-normalized embeddings, scores range from negative to positive the number of query tokens.
- On 4,874 Natural Questions passages, LateOn produced 608,414 128-dimensional token vectors requiring 311.5 MB in float32, compared with 7.5 MB for an all-MiniLM-L6-v2 dense index.
- The same multi-vector representation took 92 MB in a fast-plaid index, which stores a centroid ID and quantized residual per vector instead of the full vector.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/multi-vector-encoder)
- Source URL: https://huggingface.co/blog/multi-vector-encoder
- Ingested by Yomu: 2026-08-29T05:41:33.857Z

[Read original post](https://huggingface.co/blog/multi-vector-encoder)