NeoMME: an efficient Multimodal-native and Multilingual Encoder

Summary

NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.

Takeaways

1. 1

   NeoMME uses dynamic-resolution images, 16,384-token bidirectional context, sliding-window attention with periodic global-attention layers, and a 131k-token multilingual BPE tokenizer.

2. 2

   NeoMME-Retriever produces dense and late-interaction representations in one forward pass; dense embeddings can handle ANN retrieval before late-interaction reranking on large corpora.

3. 3

   The released model checkpoints and Hugging Face Transformers implementation are licensed under Apache 2.0, while separate dense and late-interaction checkpoints support Sentence Transformers v6 fine-tuning.