---
title: "How to Tune Batch Size: Separate Physical from Logical"
description: "A practical workflow for choosing batch size in training and inference. Learn to separate physical memory limits from logical update size, then tune with gradient accumulation, quantization, and throughput monitoring."
---

# How to Tune Batch Size: Separate Physical from Logical

**Type:** [How to](https://yomu.fyi/visual-explainers/category/how-to.md)

![A numbered how-to flow for tuning batch size. Steps: define goal, measure memory, set physical batch, choose logical batch, bridge with accumulation, monitor and adjust. A side panel contrasts physical and logical batch sizes.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/9452be74-871b-4e0e-97fa-86c704b5c7b6/image-6566cfdd-0b9b-4c80-abc4-f5ccd6dcb438.webp)

> A practical workflow for choosing batch size in training and inference. Learn to separate physical memory limits from logical update size, then tune with gradient accumulation, quantization, and throughput monitoring.

## Explanation

Batch size is not one number but two: the physical batch that fits in device memory and the logical batch that shapes the learning update. This explainer walks through a practical tuning workflow. Start by measuring memory headroom, set the physical batch, choose a logical batch for convergence or throughput, then bridge them with gradient accumulation or continuous batching. Monitor results and adjust with quantization, speculative decoding, or mixture-of-experts routing. The goal is to stop guessing and start tuning with a repeatable process.

## FAQ

### Why decouple physical and logical batch sizes?

Physical batch is limited by device memory, while logical batch controls the effective gradient update. Decoupling lets you fit a large logical batch without exceeding memory, as shown in recent masked diffusion work that removed batch size tuning.

### When should I use gradient accumulation instead of increasing batch size?

Use gradient accumulation when memory is the bottleneck and you need a larger logical batch. It simulates a larger batch by accumulating gradients over multiple forward and backward passes before updating weights.

### How do quantization and speculative decoding affect batch size choices?

Quantization reduces memory per parameter, allowing larger physical batches. Speculative decoding increases throughput per request, which can shift the optimal batch size for latency-sensitive serving. Revisit both after setting a baseline.

**Tags:** [Batch Size](https://yomu.fyi/topic/batch-size.md), [GPU](https://yomu.fyi/topic/gpu.md), [Hyperparameter Tuning](https://yomu.fyi/topic/hyperparameter-tuning.md), [Inference](https://yomu.fyi/topic/inference.md), [Model Serving](https://yomu.fyi/topic/model-serving.md), [Quantization](https://yomu.fyi/topic/quantization.md)

## Related reading

### [Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM](https://yomu.fyi/post/advancing-generative-recommendation-inference-our-collaboration-with-n.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Oct 16, 2026
- Type: Problem & solution

Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.


### [How our AI Research team uses AI Runtime](https://yomu.fyi/post/how-our-ai-research-team-uses-ai-runtime.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ethan Tang, Ivan Zhou, Irene Dea, Steven Chen, Bruce Fontaine, Ben Hansen
- Published: Oct 16, 2026
- Type: Problem & solution

Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.


### [How FactSet cut model training and deployment from days to hours with Databricks AI Runtime](https://yomu.fyi/post/how-factset-cut-model-training-and-deployment-from-days-to-hours-with.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nikhil Hiriyur Sunderraj, Dakshina Murthy Malladi, Nikitha Goram, Ryan Roser, Kate Stepp, Julia Powell, Tejas Sundaresan, Lucas de Paula
- Published: Oct 15, 2026
- Type: Problem & solution

FactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.


### [Scale Agentic Coding Workflows with Meta’s Muse Spark Open on NVIDIA GB300 NVL72](https://yomu.fyi/post/scale-agentic-coding-workflows-with-meta-s-muse-spark-open-on-nvidia-g.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Oct 15, 2026
- Type: Announcement

Meta launches Muse Spark Open, an 816B-parameter mixture-of-experts model with 42B active parameters and a 1M token context window, trained on NVIDIA GB300 NVL72 for agentic coding workflows. It plans, acts, observes, and self-corrects across multistep tasks like multifile refactoring, using tool calls and subagent coordination while holding an entire project in context without breaking it into pieces. Its 1.8B perception encoder natively processes interleaved text, images, video frames, and audio in a single pass. On GB300 NVL72 with speculative decoding, it achieves over 50K tokens per second per GPU and over 600 tokens per second per user. Sparse MoE routing activates only 42B parameters per token, and NVFP4 quantization reduces memory footprint for larger KV cache. The model is available via Hugging Face and NVIDIA NIM, with support from SGLang, vLLM, and TensorRT LLM for serving, and fine-tuning through NeMo AutoModel and Megatron Bridge.


### [Optimized support for Open NVIDIA Nemotron 3.5 Super VL](https://yomu.fyi/post/optimized-support-for-open-nvidia-nemotron-3-5-super-vl.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Eldar Kurtić, Helen Zhao, Charles Hernandez, Dipika Sikka
- Published: Oct 15, 2026
- Type: Announcement

Red Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math\_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level.


### [NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning](https://yomu.fyi/post/nvidia-nemotron-3-5-super-vl-powers-specialized-agents-with-multimodal.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Oct 15, 2026
- Type: Announcement

NVIDIA released Nemotron 3.5 Super VL, an open 120B-parameter multimodal model for specialized enterprise agents. Its mixture-of-experts architecture activates 12B parameters per token, balancing capacity and inference efficiency. The model combines reasoning, instruction following, tool use, and understanding of text, images, documents, and video for tasks including coding, web research, document intelligence, and cybersecurity. On a single GPU, it achieves up to 5x higher throughput for agentic tasks than competing open models, and it scores higher than the larger Nemotron 3 Ultra on Hermes Bench and LangChain evaluations. Early adopters report cost reductions of 40x for credit processing and 10x lower latency for receipt parsing, and organizations can post-train the model with NVIDIA NeMo for domain-specific workflows and deploy it with NVIDIA NIM and Dynamo, which serves up to 8x more users with 85% lower first-token latency.


### [The Design Space of Tri-Modal Masked Diffusion Models](https://yomu.fyi/post/the-design-space-of-tri-modal-masked-diffusion-models.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 14, 2026
- Type: Announcement

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, but prior approaches typically initialize a unimodal model and fine-tune for bimodal generation. This work diverges from that paradigm by introducing the first tri-modal Masked Diffusion Model pretrained from scratch jointly on text, image-text, and audio-text data, and systematically analyzing multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. A key contribution is a stochastic differential equation reparameterization that decouples physical and logical batch sizes, removing the need for batch size tuning reported in recent work. The authors pretrain a 3B-parameter model on 6.4 trillion tokens and provide optimized inference sampling defaults, achieving strong results on text generation, text-to-image, and text-to-speech tasks. This work represents the largest open systematic study of multimodal discrete diffusion models conducted to date.