Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Summary

Sentence Transformers v6.0 adds MultiVectorEncoder, a fourth model type for ColBERT-style late-interaction retrieval, alongside a training approach for finetuning existing models or building them from base transformers. Multi-vector models retain one small vector per token and score query-document pairs with MaxSim, where each query token selects its best document-token match and the scores are summed. The guide covers models, datasets, loss functions, training arguments, evaluators, trainers, callbacks, and multi-dataset training, with examples using pip install -U "sentence-transformers[train]". On a medical retrieval evaluation, the finetuned multi-vector-encoder/mLateOn-medical model, trained for 14.5 hours on a single RTX 3090, outperformed every general-purpose retrieval model tested across dense, sparse, lexical, and multi-vector systems.

Takeaways

1. 1

   Multi-vector retrieval uses token-level embeddings and the MaxSim operator rather than compressing each text into one vector.

2. 2

   The mLateOn-medical model was trained in 14.5 hours on a single RTX 3090 and outperformed the general-purpose retrieval models tested on the medical evaluation.

3. 3

   A 1-bit PLAID index with all vectors was 3.37 GB versus 45 GB for raw embeddings, with NDCG@10 of 0.8984; pruning reduced it to 1.45 GB at 0.8642.