---
title: "Multimodal"
description: "18 posts about Multimodal, summarised, each linking to the original."
---

# Multimodal
> 18 posts about Multimodal, summarised, each linking to the original.

## Articles

### [NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning](https://yomu.fyi/post/nvidia-nemotron-3-5-super-vl-powers-specialized-agents-with-multimodal.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Oct 15, 2026
- Type: Announcement

NVIDIA released Nemotron 3.5 Super VL, an open 120B-parameter multimodal model for specialized enterprise agents. Its mixture-of-experts architecture activates 12B parameters per token, balancing capacity and inference efficiency. The model combines reasoning, instruction following, tool use, and understanding of text, images, documents, and video for tasks including coding, web research, document intelligence, and cybersecurity. On a single GPU, it achieves up to 5x higher throughput for agentic tasks than competing open models, and it scores higher than the larger Nemotron 3 Ultra on Hermes Bench and LangChain evaluations. Early adopters report cost reductions of 40x for credit processing and 10x lower latency for receipt parsing, and organizations can post-train the model with NVIDIA NeMo for domain-specific workflows and deploy it with NVIDIA NIM and Dynamo, which serves up to 8x more users with 85% lower first-token latency.


### [Apertus 1.5 Release Notes](https://yomu.fyi/post/apertus-1-5-release-notes.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Hanna Yukhymenko, Oleg Lavrovsky, Y Metz
- Published: Oct 15, 2026
- Type: Announcement

Apertus 1.5 extends the 8B and 70B open-weight models with image and audio understanding, a thinking mode, and a 262,144-token context window, released under Apache 2.0. The models are a continued pretraining of Apertus 1.0, adding 4 trillion tokens to the 8B model and 2 trillion to the 70B model, for totals of 19T and 17T respectively. Multimodality uses an early-fusion tokenizer with Emu3.5 for images and WavTokenizer for audio, keeping the architecture unchanged and training at text-only speed. Post-training includes SFT, long-context SFT, reinforcement learning with verifiable rewards, DPO, and charter alignment via SDPO, with all data and recipes publicly documented. Evaluation results show competitive performance on reasoning, coding, and multilingual tasks, with the 8B model matching GPT-OSS-20B on translation and outperforming Claude Haiku 4.5, reported transparently via the EvalEval Coalition’s EEE schema.


### [Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents](https://yomu.fyi/post/selection-based-structured-reasoning-toward-efficient-multimodal-searc.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 15, 2026
- Type: Problem & solution

Multimodal search agents typically rely on freeform reasoning before acting, but small models often produce lengthy reasoning that offers little guidance and increases inference cost. The authors propose Selection-Based Structured Reasoning (SSR), which reframes reasoning as selecting from pre-specified natural-language candidates rather than open-ended generation. SSR enables parallel scoring by using teacher-forced prefilling and a shared context KV cache to compute token likelihoods across candidates simultaneously. Evaluated on seven multimodal search benchmarks with 2B and 4B models, SSR maintains task performance under various reinforcement learning objectives and supervised fine-tuning. The framework achieves search success rates competitive with leading agents of the same scale while reducing per-turn reasoning latency by over 90% and total per-question inference latency by 28–54%.


### [Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta](https://yomu.fyi/post/welcome-muse-spark-open-open-weight-frontier-multimodal-mixture-of-exp.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Pedro Cuenca, merve, ben burtenshaw, Guarin Flück, Quentin Gallouédec, Alvaro Bartolome, Serveurperso, Jun Kim
- Published: Oct 15, 2026
- Type: Announcement

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with approximately 816 billion total parameters and around 42 billion active per token. It processes text, images, audio, and video inputs while generating text output, supporting a context window of up to one million tokens. The model uses 256 experts per MoE layer with 8 selected per token, and the BF16 checkpoint requires about 1.5 TB of storage. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. The post provides recipes for running the model using transformers, llama.cpp, vLLM, and SGLang, as well as fine-tuning with LoRA on 64 H100 GPUs.


### [The Design Space of Tri-Modal Masked Diffusion Models](https://yomu.fyi/post/the-design-space-of-tri-modal-masked-diffusion-models.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 14, 2026
- Type: Announcement

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.


### [EmbeddingGemma 2: The Developer Guide](https://yomu.fyi/post/embeddinggemma-2-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Maarten Grootendorst, Ian Ballantyne
- Published: Oct 6, 2026
- Type: Tutorial

Modern search and RAG applications need to work across diverse content types, but finding models with strong retrieval accuracy and low latency without massive compute is challenging. EmbeddingGemma 2 is a compact open model under Apache 2.0, based on Gemma 4, that maps text, code, images, video, and audio into a unified 768-dimensional space with a modular architecture scaling from 270M to 740M parameters. Key capabilities include native multimodal retrieval, superior code understanding, modular memory footprint, and flexible vector storage via Matryoshka Representation Learning enabling dynamic truncation from 768 to 128 dimensions. Under the hood, it uses modular encoders—a text/code decoder, vision encoder, and audio encoder—that project into a shared space and can be loaded selectively to minimize memory. The developer guide demonstrates how to use sentence-transformers to load the model, embed with task prompts, handle media inputs, truncate dimensions, and choose configurations based on storage and recall needs.


### [Bring multimodal semantic search to the edge with EmbeddingGemma 2](https://yomu.fyi/post/bring-multimodal-semantic-search-to-the-edge-with-embeddinggemma-2.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Google AI Edge Team, Android ML Team
- Published: Oct 6, 2026
- Type: Announcement

Google DeepMind launched EmbeddingGemma 2, an open-weight multimodal embedding model that maps text, images, video frames, and audio into a single vector space. Designed for local, privacy-first applications, it runs on as little as ~191MB active RAM for text-only weights and ~567MB for the full multimodal model on a Google Pixel 11 Pro. The model powers two new interactive showcases in Google AI Edge Gallery: Instant Media Search, which retrieves local images and videos via natural language or example images, and Video Moments Finder, which locates specific visual moments in local video files without transcribing audio. Google AI Edge Foresight for Mac uses EmbeddingGemma 2 alongside Gemma 4 for local meeting note-taking, cross-modal retrieval, and live question answering. Optimizations include modular encoders, quantization-aware training to INT4/INT8, and Matryoshka Representation Learning slicing to reduce storage footprints by up to 8x.


### [AI Search is now generally available](https://yomu.fyi/post/ai-search-is-now-generally-available.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Ashish Vinodkumar
- Published: Oct 1, 2026
- Type: Announcement

Cloudflare’s AI Search, a managed pipeline combining Workers AI, Vectorize, R2, and Browser Run, is now generally available after a year of preview. The update adds native image embeddings via the Qwen3-VL-Embedding model with Matryoshka Representation Learning, enabling visual search by texture and composition alongside caption-based text search. AI Search now supports files up to 10 MiB and optical character recognition for scanned PDFs; queries undergo optional rewriting, parallel vector and keyword search, fusion, and optional reranking. Previously reliant on caption generation only, image retrieval now embeds query images directly using multimodal models, while text-only models convert images to text via ToMarkdown. Billing starts November 1, 2026, with a free monthly allotment of 5 million ingestion tokens, 10 GB storage, 1,000 semantic queries, and 1,000 full-text queries, with no capacity units or monthly minimums.


### [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://yomu.fyi/post/putting-captions-to-the-test-evaluating-video-caption-quality-through.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 11, 2026
- Type: Benchmark

Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance.


### [SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign](https://yomu.fyi/post/simpledesign-a-joint-model-for-protein-sequence-and-structure-codesign.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 11, 2026
- Type: Problem & solution

SimpleDesign addresses protein design as a joint modeling problem involving amino acid sequences and three-dimensional structures, whose interplay determines protein function. Unlike approaches that first train autoencoders and then model their latent representations, it uses a single-stage, end-to-end objective directly in data space. The objective combines discrete cross-entropy for sequences with a regression objective for structures, while Transformer-based multimodal backbones provide modality-specific processing and global self-attention across both modalities. Trained on more than 2M sequence-structure pairs, SimpleDesign achieves competitive performance on co-design and unconditional sequence and structure generation benchmarks. The work argues, based on these results, that multi-stage training is not necessary for performant co-design models.


### [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving](https://yomu.fyi/post/when-to-use-encode-prefill-decode-disaggregation-to-accelerate-multimo.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 9, 2026
- Type: Explainer

Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.


### [NeoMME: an efficient Multimodal-native and Multilingual Encoder](https://yomu.fyi/post/neomme-an-efficient-multimodal-native-and-multilingual-encoder.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tony Wu, Aurélien Lac
- Published: Sep 3, 2026
- Type: Explainer

NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.


### [Together AI brings Thinking Machines Lab’s new model Inkling on day 0](https://yomu.fyi/post/together-ai-brings-thinking-machines-lab-s-new-model-inkling-on-day-0.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 15, 2026
- Type: Announcement

Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.


### [Building Production-Scale Visual AI for Claims and Returns at Swiggy Instamart](https://yomu.fyi/post/building-production-scale-visual-ai-for-claims-and-returns-at-swiggy-i.md)
- Company: [Swiggy](https://yomu.fyi/company/swiggy.md)
- Author: Harshit Juneja
- Published: Jun 16, 2026
- Type: Problem & solution

Swiggy Instamart built two vision-language-model systems to handle more than 40,000 daily claims involving wrong, expired, poor-quality, damaged, and packaging-related items, while reducing reliance on human verification. The Image Validation Model (IVM) compares customer uploads with catalog images, routes each category through a specialised LangGraph StateGraph handler, and combines its structured verdict with fraud, confidence, and business-rule signals. Reverse Quality Check (RQC) compares delivery executives’ pickup photos with the original claim images, allowing matching returns to proceed and prompting retries when they do not. The architecture keeps visual assessment probabilistic but makes routing, aggregation, escalation, and financial decisions deterministic, with traces for category, outputs, confidence, cost, and latency. After more than six months in production, the systems are associated with a 6% customer-satisfaction increase, 20,000 fewer weekly Image Validation tickets, 98.5% RQC accuracy, and over 5,000 daily RQC tickets resolved without human intervention.


### [Gemma 4 12B: The Developer Guide](https://yomu.fyi/post/gemma-4-12b-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: André Susano Pinto, Andreas Steiner, Karolis Misiunas, Karsten Roth, Michael Tschannen, Omar Sanseviero
- Published: Jun 3, 2026
- Type: Announcement

Gemma 4 12B is a dense multimodal model with a unified, encoder-free architecture, native audio input, and a size intended for local execution on dedicated GPU laptops with 16GB VRAM or unified memory. Instead of separate vision and audio encoders, it uses a 35M-parameter vision embedder that projects raw 48x48 patches and a linear projection for raw 16 kHz audio sliced into 40ms frames; both feed a decoder-only transformer with shared weights. That design reduces multimodal latency and fragmented memory footprints, while allowing downstream adapters such as LoRA or full tuning to update the multimodal token loop in one pass. Capabilities include automatic speech recognition, agentic reasoning, diarization, video understanding, and coding, with examples covering a locally built image-processing app and five minutes of video processed at 1 FPS with audio; LiteRT-LM adds offline macOS apps, sandboxed Python execution, and an OpenAI-compatible local API server with stateless prefix caching.


### [Boosting multimodal inference performance by >10% with a single Python dictionary](https://yomu.fyi/post/boosting-multimodal-inference-performance-by-10-with-a-single-python-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 4, 2026
- Type: Problem & solution

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.


### [Build Better AI Agents: 5 Developer Tips from the Agent Bake-Off](https://yomu.fyi/post/build-better-ai-agents-5-developer-tips-from-the-agent-bake-off.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Frank Guan, Abraham Gomez
- Published: Apr 14, 2026
- Type: Tutorial

The Google Cloud AI Agent Bake-Off is presented as a test of building autonomous agents under severe time constraints for e-commerce, banking modernization, and startup go-to-market problems. Its five patterns recommend decomposing monolithic prompts into specialized sub-agents under a supervisor, designing modularly because model capabilities can quickly replace custom harnesses, and treating multimodality as native. It also advocates open protocols—including MCP, A2A, UCP, AP2, A2UI, and AG-UI—for agent and tool connections instead of bespoke wrappers. Finally, it separates probabilistic reasoning from execution: strict JSON schemas such as Pydantic validate LLM outputs before deterministic Python functions or SQL perform calculations and writes. The stated conclusion is that reliable production agents depend on rigorous software architecture rather than elaborate prompts or flashy single-shot demos.


### [Machine Learning in Content Moderation at Etsy](https://yomu.fyi/post/machine-learning-in-content-moderation-at-etsy.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: David Azcona
- Published: Aug 6, 2024
- Type: Explainer

Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services.