Red HatBoaz Ben ShabatProblem & solution
Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.
The author ran experiments on a CoreWeave Kubernetes cluster with 16 NVIDIA H200 GPUs (140 GB HBM3e each) across two nodes connected by RDMA/InfiniBand. They used vLLM v0.18.0 with Red Hat AI Inference 3.4 and llm-d. The workload was enterprise RAG-style: 2,000-token inputs, 100-token outputs, 100 concurrent users, and 50% prefix cache hit rate. Two models were tested: Qwen3-32B (BF16) and Meta-Llama-3.1-70B (FP8).
The author systematically optimized inference by testing aggregated and prefill-decode (PD) disaggregated architectures, sweeping tensor parallelism (TP) values (TP1 to TP8), and finding the optimal prefill-to-decode GPU split (e.g., 3P+1D TP4 for default). They tuned engine parameters like max_num_seqs using a four-factor formula accounting for activation memory, KV cache capacity, concurrency, and scheduler caps, and adjusted block_size for prefix cache alignment. They also optimized request routing with custom endpoint picker (EPP) weights derived from Prometheus metrics. The entire process was automated by the ServeIt Studio tool, which runs a 12-step pipeline on Kubernetes.
Default inference configurations are conservative to work on any hardware; on 16 H200 GPUs, default aggregated TTFT was 995ms, while tuning reduced it to 304ms for the same hardware and model (Qwen3-32B).
Prefill-decode (PD) disaggregation can improve TTFT by separating compute-bound prefill from memory-bound decode, but the optimal split depends on workload; more prefill pods reduce first-token latency for long inputs.
Engine parameters such as max_num_seqs and block_size must be calibrated per hardware; the author derived max_num_seqs from activation memory, KV cache, concurrency, and scheduler caps, and aligned block_size with prefix cache hit patterns.
PD disaggregation splits GPU pods into prefill (input processing) and decode (token generation) groups. Each type is optimized for its task — prefill is compute-bound and decode is memory-bound — allowing better hardware utilization. It requires RDMA for KV cache transfer between pods.
Yes, the author achieved 71% TTFT reduction on the same hardware by tuning configuration only. Default settings are conservative; matching parameters like TP, PD split, max_num_seqs, and block_size to your specific hardware and workload can unlock significant performance gains.
The author calibrated TP by isolating decode (ISL=1) and prefill (OSL=1) and measuring latency for each TP degree. The best TP balances prefill and decode performance; for Qwen3-32B, 2xTP8 was best for aggregated mode.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
bfcache freezes page snapshots for instant back/forward navigation, but unload listeners, cache headers, or third-party scripts can silently disable it. This explainer maps the trade-offs and the metrics that reveal them.
The API gateway is the single boundary where protocol translation, auth, quotas, and routing become shared policy instead of per-service code. This layered diagram shows what belongs there and why.
Streaming and batch ASR pipelines compared: how input buffer length, speaker diarization, and where audio is processed trade latency against transcript quality and privacy.
Antigravity SDK's hybrid architecture separates planning from execution: a cloud model plans, local models run most tokens on-device. See how routing, execution, and the data boundary fit together.
A walkthrough of how a 3D CT model turns a voxel volume into a structured diagnostic report: isotropic resampling, 3D patch tokens, spatial encoding, and language-model reasoning.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
