Multimodal Embedding & Reranker Models with Sentence Transformers

Summary

Sentence Transformers v5.4 adds multimodal embedding and reranking support for text, images, audio, and video through its Python API. Embedding models map modalities into a shared space, while multimodal Cross Encoder rerankers score relevance for mixed-modality pairs, enabling cross-modal similarity, retrieval, and multimodal RAG workflows. The guide shows model loading, URL-based image encoding, encode_query() and encode_document() prompt handling, similarity calculation, and ranking, with a Qwen3-VL-Embedding-2B example producing 2,048-dimensional image vectors and matching text queries to images. It notes that cross-modal scores can be lower because of the modality gap, while relative ordering can still support retrieval, and that rerankers generally offer higher quality at the cost of slower per-pair processing; installation extras and GPU memory requirements vary by modality and model.

Takeaways

1. 1

   Multimodal embedding models place text, images, audio, and video in a shared embedding space, allowing cross-modal similarity comparisons such as text queries against image documents.

2. 2

   encode_query() and encode_document() automatically select configured query or document prompts and otherwise act as wrappers around encode().

3. 3

   Multimodal rerankers can provide higher quality than embedding models but process each pair individually, making them slower; VLM-based models also require substantial GPU memory.