---
title: "Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers"
description: "Sentence Transformers provides a Python library for training and using embedding and reranker models across text, images, audio, and video in retrieval and semantic search applications. The post shows how to finetune multimodal models on task-specific data, using Visual Document Retrieval as the example: matching text queries with document page images while preserving charts, tables, and layout. It describes the SentenceTransformerTrainer pipeline, model and processor configuration, dataset handling, and alternative Router-based encoders, while also covering multimodal reranker training with BinaryCrossEntropyLoss and direction-specific datasets. On its evaluation data, finetuning Qwen/Qwen3-VL-Embedding-2B raised NDCG@10 from 0.888 to 0.947 and outperformed all tested VDR models, including models up to 4x its size."
---

# Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tom Aarsen · Apr 16, 2026

**Type:** Tutorial

## Summary

Sentence Transformers provides a Python library for training and using embedding and reranker models across text, images, audio, and video in retrieval and semantic search applications. The post shows how to finetune multimodal models on task-specific data, using Visual Document Retrieval as the example: matching text queries with document page images while preserving charts, tables, and layout. It describes the SentenceTransformerTrainer pipeline, model and processor configuration, dataset handling, and alternative Router-based encoders, while also covering multimodal reranker training with BinaryCrossEntropyLoss and direction-specific datasets. On its evaluation data, finetuning Qwen/Qwen3-VL-Embedding-2B raised NDCG@10 from 0.888 to 0.947 and outperformed all tested VDR models, including models up to 4x its size.

## Context

General-purpose multimodal embedding models are trained across diverse languages and tasks, so they are rarely optimal for a specific task. Visual Document Retrieval requires matching text queries to relevant document screenshots while understanding document layouts, charts, tables, and text.

## Approach / What changed

The post finetunes Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval using SentenceTransformerTrainer, a task-specific dataset, configurable processor and model arguments, and multimodal training components. It also describes Router-based models with separate modality encoders and covers multimodal reranker training with direction-specific datasets and BinaryCrossEntropyLoss.

## Takeaways

- Finetuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval improved evaluation NDCG@10 from 0.888 to 0.947.
- The preprocessed training dataset contains 53,512 English query-image samples and resolves four hard negatives per sample into document screenshot images.
- Router-based multimodal models use separate encoders whose embedding spaces must be aligned through training; a Dense projection can map them into a shared space.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/train-multimodal-sentence-transformers)
- Source URL: https://huggingface.co/blog/train-multimodal-sentence-transformers
- Ingested by Yomu: 2026-08-29T01:47:49.035Z

[Read original post](https://huggingface.co/blog/train-multimodal-sentence-transformers)