---
title: "Multimodal Embedding & Reranker Models with Sentence Transformers"
description: "Sentence Transformers v5.4 adds multimodal embedding and reranking support for text, images, audio, and video through its Python API. Embedding models map modalities into a shared space, while multimodal Cross Encoder rerankers score relevance for mixed-modality pairs, enabling cross-modal similarity, retrieval, and multimodal RAG workflows. The guide shows model loading, URL-based image encoding, encode_query() and encode_document() prompt handling, similarity calculation, and ranking, with a Qwen3-VL-Embedding-2B example producing 2,048-dimensional image vectors and matching text queries to images. It notes that cross-modal scores can be lower because of the modality gap, while relative ordering can still support retrieval, and that rerankers generally offer higher quality at the cost of slower per-pair processing; installation extras and GPU memory requirements vary by modality and model."
---

# Multimodal Embedding & Reranker Models with Sentence Transformers

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tom Aarsen · Apr 9, 2026

**Type:** Tutorial

## Summary

Sentence Transformers v5.4 adds multimodal embedding and reranking support for text, images, audio, and video through its Python API. Embedding models map modalities into a shared space, while multimodal Cross Encoder rerankers score relevance for mixed-modality pairs, enabling cross-modal similarity, retrieval, and multimodal RAG workflows. The guide shows model loading, URL-based image encoding, encode\_query() and encode\_document() prompt handling, similarity calculation, and ranking, with a Qwen3-VL-Embedding-2B example producing 2,048-dimensional image vectors and matching text queries to images. It notes that cross-modal scores can be lower because of the modality gap, while relative ordering can still support retrieval, and that rerankers generally offer higher quality at the cost of slower per-pair processing; installation extras and GPU memory requirements vary by modality and model.

## Context

Sentence Transformers traditionally handled text embedding and reranking, while multimodal applications require comparing or scoring inputs across text, images, audio, and video. The update addresses use cases including visual document retrieval, cross-modal search, and multimodal RAG pipelines.

## Approach / What changed

Use Sentence Transformers v5.4 multimodal models through the existing API: install modality-specific extras, load a model, encode mixed inputs, compare embeddings, and use encode\_query() and encode\_document() for retrieval prompts. Apply multimodal Cross Encoder rerankers to score candidate pairs, with processor and model settings available for configuration.

## Takeaways

- Multimodal embedding models place text, images, audio, and video in a shared embedding space, allowing cross-modal similarity comparisons such as text queries against image documents.
- encode\_query() and encode\_document() automatically select configured query or document prompts and otherwise act as wrappers around encode().
- Multimodal rerankers can provide higher quality than embedding models but process each pair individually, making them slower; VLM-based models also require substantial GPU memory.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/multimodal-sentence-transformers)
- Source URL: https://huggingface.co/blog/multimodal-sentence-transformers
- Ingested by Yomu: 2026-08-29T01:47:31.209Z

[Read original post](https://huggingface.co/blog/multimodal-sentence-transformers)