Welcome EmbeddingGemma, Google's new efficient embedding model

Summary

Google DeepMind released EmbeddingGemma, a multilingual embedding model with 308 million parameters and a 2048-token context window designed for on-device applications. Based on the Gemma3 transformer backbone, the architecture replaces causal attention with bidirectional attention to function as an encoder, followed by mean pooling and two dense layers producing 768-dimensional vectors. The model incorporates Matryoshka Representation Learning, allowing outputs to be truncated down to 512, 256, or 128 dimensions for reduced memory and storage footprints. Trained on approximately 320 billion multilingual tokens across more than 100 languages, the quantized model operates under 200 MB of RAM. In domain-specific evaluations on the MIRIAD dataset, fine-tuning increased NDCG@10 from 0.8340 to 0.8862, outperforming larger baselines.

Takeaways

1. 1

   EmbeddingGemma contains 308 million parameters, supports a 2048-token context window across more than 100 languages, and consumes under 200 MB of RAM when quantized.

2. 2

   The architecture modifies Gemma3 from a causal decoder to a bidirectional encoder, using mean pooling and dense layers to yield 768-dimensional embeddings truncatable to 512, 256, or 128 dimensions via Matryoshka Representation Learning.

3. 3

   Fine-tuning EmbeddingGemma on the MIRIAD medical retrieval dataset boosted retrieval performance from 0.8340 to 0.8862 NDCG@10, exceeding the performance of multiple larger baseline models on the task.