---
title: "Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers"
description: "Sentence Transformers v6.0 adds MultiVectorEncoder, a fourth model type for ColBERT-style late-interaction retrieval, alongside a training approach for finetuning existing models or building them from base transformers. Multi-vector models retain one small vector per token and score query-document pairs with MaxSim, where each query token selects its best document-token match and the scores are summed. The guide covers models, datasets, loss functions, training arguments, evaluators, trainers, callbacks, and multi-dataset training, with examples using pip install -U \"sentence-transformers[train]\". On a medical retrieval evaluation, the finetuned multi-vector-encoder/mLateOn-medical model, trained for 14.5 hours on a single RTX 3090, outperformed every general-purpose retrieval model tested across dense, sparse, lexical, and multi-vector systems."
---

# Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tom Aarsen · Aug 26, 2026

**Type:** Tutorial

## Summary

Sentence Transformers v6.0 adds MultiVectorEncoder, a fourth model type for ColBERT-style late-interaction retrieval, alongside a training approach for finetuning existing models or building them from base transformers. Multi-vector models retain one small vector per token and score query-document pairs with MaxSim, where each query token selects its best document-token match and the scores are summed. The guide covers models, datasets, loss functions, training arguments, evaluators, trainers, callbacks, and multi-dataset training, with examples using pip install -U "sentence-transformers\[train\]". On a medical retrieval evaluation, the finetuned multi-vector-encoder/mLateOn-medical model, trained for 14.5 hours on a single RTX 3090, outperformed every general-purpose retrieval model tested across dense, sparse, lexical, and multi-vector systems.

## Context

General-purpose retrieval models may miss domain-specific vocabulary, query styles, and relevance signals. The post also reports that released models often truncate documents at 180 to 512 tokens, while the evaluated medical passages averaged 941 tokens; this truncation cost up to 0.24 NDCG@10 in the medical evaluation.

## Approach / What changed

Use Sentence Transformers v6.0's MultiVectorEncoder with domain data, a suitable loss, training arguments, evaluators, and trainer components. The post covers finetuning an existing checkpoint or adding a token-level projection to a base transformer, configuring document length, and using a punctuation skiplist. It also evaluates token pooling, PLAID residual quantization, and pruning for index size reduction.

## Takeaways

- Multi-vector retrieval uses token-level embeddings and the MaxSim operator rather than compressing each text into one vector.
- The mLateOn-medical model was trained in 14.5 hours on a single RTX 3090 and outperformed the general-purpose retrieval models tested on the medical evaluation.
- A 1-bit PLAID index with all vectors was 3.37 GB versus 45 GB for raw embeddings, with NDCG@10 of 0.8984; pruning reduced it to 1.45 GB at 0.8642.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/train-multi-vector-encoder)
- Source URL: https://huggingface.co/blog/train-multi-vector-encoder
- Ingested by Yomu: 2026-08-29T05:42:12.171Z

[Read original post](https://huggingface.co/blog/train-multi-vector-encoder)