Loading…
Topic
14 posts about Multimodal. Every summary links to the original.
See the idea
A walkthrough of how a 3D CT model turns a voxel volume into a structured diagnostic report: isotropic resampling, 3D patch tokens, spatial encoding, and language-model reasoning.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.
Modern search and RAG applications need to work across diverse content types, but finding models with strong retrieval accuracy and low latency without massive compute is challenging. EmbeddingGemma 2 is a compact open model under Apache 2.0, based on Gemma 4, that maps text, code, images, video, and audio into a unified 768-dimensional space with a modular architecture scaling from 270M to 740M parameters. Key capabilities include native multimodal retrieval, superior code understanding, modular memory footprint, and flexible vector storage via Matryoshka Representation Learning enabling dynamic truncation from 768 to 128 dimensions. Under the hood, it uses modular encoders—a text/code decoder, vision encoder, and audio encoder—that project into a shared space and can be loaded selectively to minimize memory. The developer guide demonstrates how to use sentence-transformers to load the model, embed with task prompts, handle media inputs, truncate dimensions, and choose configurations based on storage and recall needs.
Google DeepMind launched EmbeddingGemma 2, an open-weight multimodal embedding model that maps text, images, video frames, and audio into a single vector space. Designed for local, privacy-first applications, it runs on as little as ~191MB active RAM for text-only weights and ~567MB for the full multimodal model on a Google Pixel 11 Pro. The model powers two new interactive showcases in Google AI Edge Gallery: Instant Media Search, which retrieves local images and videos via natural language or example images, and Video Moments Finder, which locates specific visual moments in local video files without transcribing audio. Google AI Edge Foresight for Mac uses EmbeddingGemma 2 alongside Gemma 4 for local meeting note-taking, cross-modal retrieval, and live question answering. Optimizations include modular encoders, quantization-aware training to INT4/INT8, and Matryoshka Representation Learning slicing to reduce storage footprints by up to 8x.
Cloudflare’s AI Search, a managed pipeline combining Workers AI, Vectorize, R2, and Browser Run, is now generally available after a year of preview. The update adds native image embeddings via the Qwen3-VL-Embedding model with Matryoshka Representation Learning, enabling visual search by texture and composition alongside caption-based text search. AI Search now supports files up to 10 MiB and optical character recognition for scanned PDFs; queries undergo optional rewriting, parallel vector and keyword search, fusion, and optional reranking. Previously reliant on caption generation only, image retrieval now embeds query images directly using multimodal models, while text-only models convert images to text via ToMarkdown. Billing starts November 1, 2026, with a free monthly allotment of 5 million ingestion tokens, 10 GB storage, 1,000 semantic queries, and 1,000 full-text queries, with no capacity units or monthly minimums.
Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance.
SimpleDesign addresses protein design as a joint modeling problem involving amino acid sequences and three-dimensional structures, whose interplay determines protein function. Unlike approaches that first train autoencoders and then model their latent representations, it uses a single-stage, end-to-end objective directly in data space. The objective combines discrete cross-entropy for sequences with a regression objective for structures, while Transformer-based multimodal backbones provide modality-specific processing and global self-attention across both modalities. Trained on more than 2M sequence-structure pairs, SimpleDesign achieves competitive performance on co-design and unconditional sequence and structure generation benchmarks. The work argues, based on these results, that multi-stage training is not necessary for performant co-design models.
Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.
NeoMME is a family of 260M and 800M multilingual multimodal encoders designed to represent text and images without a separate pretrained vision tower or causal language model. A single bidirectional Transformer processes text tokens and raw 32×32 image patches, while pretraining from scratch uses a masked discrete-diffusion text-denoising objective with high masking to encourage image-grounded reconstruction. The retrieval variant adds jointly trained dense mean-pooled and 128-dimensional late-interaction heads, returning both in one forward pass for page-image retrieval. On ViDoRe v3, the 260M model reaches 0.523 and both sizes lie on the model-size Pareto frontier; at 2048×2048, it encodes about 51 pages per second on an NVIDIA L40S, roughly twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction storage from roughly 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10.
Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.
Swiggy Instamart built two vision-language-model systems to handle more than 40,000 daily claims involving wrong, expired, poor-quality, damaged, and packaging-related items, while reducing reliance on human verification. The Image Validation Model (IVM) compares customer uploads with catalog images, routes each category through a specialised LangGraph StateGraph handler, and combines its structured verdict with fraud, confidence, and business-rule signals. Reverse Quality Check (RQC) compares delivery executives’ pickup photos with the original claim images, allowing matching returns to proceed and prompting retries when they do not. The architecture keeps visual assessment probabilistic but makes routing, aggregation, escalation, and financial decisions deterministic, with traces for category, outputs, confidence, cost, and latency. After more than six months in production, the systems are associated with a 6% customer-satisfaction increase, 20,000 fewer weekly Image Validation tickets, 98.5% RQC accuracy, and over 5,000 daily RQC tickets resolved without human intervention.
Gemma 4 12B is a dense multimodal model with a unified, encoder-free architecture, native audio input, and a size intended for local execution on dedicated GPU laptops with 16GB VRAM or unified memory. Instead of separate vision and audio encoders, it uses a 35M-parameter vision embedder that projects raw 48x48 patches and a linear projection for raw 16 kHz audio sliced into 40ms frames; both feed a decoder-only transformer with shared weights. That design reduces multimodal latency and fragmented memory footprints, while allowing downstream adapters such as LoRA or full tuning to update the multimodal token loop in one pass. Capabilities include automatic speech recognition, agentic reasoning, diarization, video understanding, and coding, with examples covering a locally built image-processing app and five minutes of video processed at 1 FPS with audio; LiteRT-LM adds offline macOS apps, sandboxed Python execution, and an OpenAI-compatible local API server with stateless prefix caching.
Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process_input_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage._new_shared_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.
The Google Cloud AI Agent Bake-Off is presented as a test of building autonomous agents under severe time constraints for e-commerce, banking modernization, and startup go-to-market problems. Its five patterns recommend decomposing monolithic prompts into specialized sub-agents under a supervisor, designing modularly because model capabilities can quickly replace custom harnesses, and treating multimodality as native. It also advocates open protocols—including MCP, A2A, UCP, AP2, A2UI, and AG-UI—for agent and tool connections instead of bespoke wrappers. Finally, it separates probabilistic reasoning from execution: strict JSON schemas such as Pydantic validate LLM outputs before deterministic Python functions or SQL perform calculations and writes. The stated conclusion is that reliable production agents depend on rigorous software architecture rather than elaborate prompts or flashy single-shot demos.
Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services.
The best engineering reads, once a week.
