Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Summary

Sentence Transformers provides a Python library for training and using embedding and reranker models across text, images, audio, and video in retrieval and semantic search applications. The post shows how to finetune multimodal models on task-specific data, using Visual Document Retrieval as the example: matching text queries with document page images while preserving charts, tables, and layout. It describes the SentenceTransformerTrainer pipeline, model and processor configuration, dataset handling, and alternative Router-based encoders, while also covering multimodal reranker training with BinaryCrossEntropyLoss and direction-specific datasets. On its evaluation data, finetuning Qwen/Qwen3-VL-Embedding-2B raised NDCG@10 from 0.888 to 0.947 and outperformed all tested VDR models, including models up to 4x its size.

Takeaways

1. 1

   Finetuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval improved evaluation NDCG@10 from 0.888 to 0.947.

2. 2

   The preprocessed training dataset contains 53,512 English query-image samples and resolves four hard negatives per sample into document screenshot images.

3. 3

   Router-based multimodal models use separate encoders whose embedding spaces must be aligned through training; a Dense projection can map them into a shared space.