A practical workflow for choosing batch size in training and inference. Learn to separate physical memory limits from logical update size, then tune with gradient accumulation, quantization, and throughput monitoring.
Batch size is not one number but two: the physical batch that fits in device memory and the logical batch that shapes the learning update. This explainer walks through a practical tuning workflow. Start by measuring memory headroom, set the physical batch, choose a logical batch for convergence or throughput, then bridge them with gradient accumulation or continuous batching. Monitor results and adjust with quantization, speculative decoding, or mixture-of-experts routing. The goal is to stop guessing and start tuning with a repeatable process.
Physical batch is limited by device memory, while logical batch controls the effective gradient update. Decoupling lets you fit a large logical batch without exceeding memory, as shown in recent masked diffusion work that removed batch size tuning.
Use gradient accumulation when memory is the bottleneck and you need a larger logical batch. It simulates a larger batch by accumulating gradients over multiple forward and backward passes before updating weights.
Quantization reduces memory per parameter, allowing larger physical batches. Speculative decoding increases throughput per request, which can shift the optimal batch size for latency-sensitive serving. Revisit both after setting a baseline.
The posts behind it
Spotify · · Problem & solutionSpotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.
Databricks · · Problem & solutionDatabricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.
Databricks · · Problem & solutionFactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.
NVIDIA · · AnnouncementMeta launches Muse Spark Open, an 816B-parameter mixture-of-experts model with 42B active parameters and a 1M token context window, trained on NVIDIA GB300 NVL72 for agentic coding workflows. It plans, acts, observes, and self-corrects across multistep tasks like multifile refactoring, using tool calls and subagent coordination while holding an entire project in context without breaking it into pieces. Its 1.8B perception encoder natively processes interleaved text, images, video frames, and audio in a single pass. On GB300 NVL72 with speculative decoding, it achieves over 50K tokens per second per GPU and over 600 tokens per second per user. Sparse MoE routing activates only 42B parameters per token, and NVFP4 quantization reduces memory footprint for larger KV cache. The model is available via Hugging Face and NVIDIA NIM, with support from SGLang, vLLM, and TensorRT LLM for serving, and fine-tuning through NeMo AutoModel and Megatron Bridge.
Red Hat · · AnnouncementRed Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level.
NVIDIA · · AnnouncementNVIDIA released Nemotron 3.5 Super VL, an open 120B-parameter multimodal model for specialized enterprise agents. Its mixture-of-experts architecture activates 12B parameters per token, balancing capacity and inference efficiency. The model combines reasoning, instruction following, tool use, and understanding of text, images, documents, and video for tasks including coding, web research, document intelligence, and cybersecurity. On a single GPU, it achieves up to 5x higher throughput for agentic tasks than competing open models, and it scores higher than the larger Nemotron 3 Ultra on Hermes Bench and LangChain evaluations. Early adopters report cost reductions of 40x for credit processing and 10x lower latency for receipt parsing, and organizations can post-train the model with NVIDIA NeMo for domain-specific workflows and deploy it with NVIDIA NIM and Dynamo, which serves up to 8x more users with 85% lower first-token latency.
Apple · · AnnouncementDiscrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.
More like this
A five-step path from fragmented payment silos to a canonical, governed record that AI agents can act on: model the payment, unify the copy, stream events, wrap writes in transactions, and scope agent authority.
A five-step loop for treating bots as accountable actors: verify identity, honor opt-out preferences, measure sustained behavior, band trust, and bound blast radius.
A four-checkpoint how-to for agent loops: name a measurable job, bound the tool surface, refresh sandbox fixtures from real traffic, and verify at agent speed.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
