---
title: "NeoMME: an efficient Multimodal-native and Multilingual Encoder"
description: "NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10."
---

# NeoMME: an efficient Multimodal-native and Multilingual Encoder

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tony Wu, Aurélien Lac · Sep 3, 2026

**Type:** Explainer

## Summary

NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.

## Context

Many visual document retrievers adapt generative visual language models with separate pretrained vision towers, projectors, and causal decoders. The post argues that retrieval, classification, and token labeling do not require autoregressive text generation, so those components add parameter and compute overhead. NeoMME was designed to provide a more efficient bidirectional multimodal encoder.

## Approach / What changed

NeoMME is trained from scratch as a masked discrete-diffusion text denoiser using one bidirectional Transformer for multilingual text tokens and raw image patches. Its retrieval variant is fine-tuned on page images and adds dense and late-interaction heads. Hierarchical token pooling and asymmetric quantization are used to reduce late-interaction index storage.

## Takeaways

- NeoMME uses dynamic-resolution images, 16,384-token bidirectional context, sliding-window attention with periodic global-attention layers, and a 131k-token multilingual BPE tokenizer.
- NeoMME-Retriever produces dense and late-interaction representations in one forward pass; dense embeddings can handle ANN retrieval before late-interaction reranking on large corpora.
- The released model checkpoints and Hugging Face Transformers implementation are licensed under Apache 2.0, while separate dense and late-interaction checkpoints support Sentence Transformers v6 fine-tuning.

**Tags:** [AI](https://yomu.fyi/topic/ai), [Multimodal](https://yomu.fyi/topic/multimodal), [Quantization](https://yomu.fyi/topic/quantization), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/Hcompany/neomme)
- Source URL: https://huggingface.co/blog/Hcompany/neomme
- Ingested by Yomu: 2026-09-03T16:00:53.376Z

[Read original post](https://huggingface.co/blog/Hcompany/neomme)