---
title: "How I massively improved my AI inference performance without buying new hardware"
description: "Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes."
---

# How I massively improved my AI inference performance without buying new hardware

[Red Hat](https://yomu.fyi/company/red-hat) · Boaz Ben Shabat · Sep 23, 2026

**Type:** Problem & solution

## Summary

Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.

## Context

The author ran experiments on a CoreWeave Kubernetes cluster with 16 NVIDIA H200 GPUs (140 GB HBM3e each) across two nodes connected by RDMA/InfiniBand. They used vLLM v0.18.0 with Red Hat AI Inference 3.4 and llm-d. The workload was enterprise RAG-style: 2,000-token inputs, 100-token outputs, 100 concurrent users, and 50% prefix cache hit rate. Two models were tested: Qwen3-32B (BF16) and Meta-Llama-3.1-70B (FP8).

## Approach / What changed

The author systematically optimized inference by testing aggregated and prefill-decode (PD) disaggregated architectures, sweeping tensor parallelism (TP) values (TP1 to TP8), and finding the optimal prefill-to-decode GPU split (e.g., 3P+1D TP4 for default). They tuned engine parameters like max\_num\_seqs using a four-factor formula accounting for activation memory, KV cache capacity, concurrency, and scheduler caps, and adjusted block\_size for prefix cache alignment. They also optimized request routing with custom endpoint picker (EPP) weights derived from Prometheus metrics. The entire process was automated by the ServeIt Studio tool, which runs a 12-step pipeline on Kubernetes.

## Takeaways

- Default inference configurations are conservative to work on any hardware; on 16 H200 GPUs, default aggregated TTFT was 995ms, while tuning reduced it to 304ms for the same hardware and model (Qwen3-32B).
- Prefill-decode (PD) disaggregation can improve TTFT by separating compute-bound prefill from memory-bound decode, but the optimal split depends on workload; more prefill pods reduce first-token latency for long inputs.
- Engine parameters such as max\_num\_seqs and block\_size must be calibrated per hardware; the author derived max\_num\_seqs from activation memory, KV cache, concurrency, and scheduler caps, and aligned block\_size with prefix cache hit patterns.

## FAQ

### What is prefill-decode disaggregation and why does it help?

PD disaggregation splits GPU pods into prefill (input processing) and decode (token generation) groups. Each type is optimized for its task — prefill is compute-bound and decode is memory-bound — allowing better hardware utilization. It requires RDMA for KV cache transfer between pods.

### Can I achieve similar improvements without buying new hardware?

Yes, the author achieved 71% TTFT reduction on the same hardware by tuning configuration only. Default settings are conservative; matching parameters like TP, PD split, max\_num\_seqs, and block\_size to your specific hardware and workload can unlock significant performance gains.

### How do I find the optimal tensor parallelism for my setup?

The author calibrated TP by isolating decode (ISL=1) and prefill (OSL=1) and measuring latency for each TP degree. The best TP balances prefill and decode performance; for Qwen3-32B, 2xTP8 was best for aggregated mode.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/23/i-improved-my-ai-inference-performance-without-buying-new-hardware)
- Source URL: https://developers.redhat.com/articles/2026/09/23/i-improved-my-ai-inference-performance-without-buying-new-hardware
- Ingested by Yomu: 2026-09-23T15:00:12.155Z

[Read original post](https://developers.redhat.com/articles/2026/09/23/i-improved-my-ai-inference-performance-without-buying-new-hardware)