Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Summary

Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB.

Takeaways

1. 1

   MaxSim selects the highest similarity between each query token and any document token, then sums those values; with L2-normalized embeddings, scores range from negative to positive the number of query tokens.

2. 2

   On 4,874 Natural Questions passages, LateOn produced 608,414 128-dimensional token vectors requiring 311.5 MB in float32, compared with 7.5 MB for an all-MiniLM-L6-v2 dense index.

3. 3

   The same multi-vector representation took 92 MB in a fast-plaid index, which stores a centroid ID and quantized residual per vector instead of the full vector.