NVIDIAElizabeth GoodmanBenchmark
AIPerf is NVIDIA’s successor to GenAI-Perf, intended to benchmark large-scale LLM inference without letting the load client become the bottleneck. Its multiprocess architecture combines worker processes, separate record-processor services, and ZMQ coordination, while supporting 15+ endpoint types, public datasets, and trace-replay formats. The walkthrough serves Qwen3-0.6B through vLLM, using streaming synthetic requests with fixed 128-token inputs and outputs; AIPerf reports TTFT, ITL, request latency, output token throughput, percentile distributions, and optional GPU telemetry in CSV and JSON. A second profile uses Poisson arrivals averaging 10 requests per second, variable input and output lengths, and a fixed random seed, producing wider latency distributions than the static baseline as concurrency and workload variability increase.
The post provides a practical benchmark workflow for LLM inference, beginning with a fixed synthetic vLLM workload and then introducing Poisson arrivals and variable sequence lengths. It explains how to interpret latency, throughput, percentile, and GPU measurements from AIPerf runs.
AIPerf uses a multiprocess load-generation architecture with worker processes, separate record-processor services, and ZMQ coordination. The walkthrough runs Qwen3-0.6B on vLLM with streaming synthetic traffic, then changes arrival patterns, token-length distributions, and request rate to model more variable workloads. A fixed random seed supports reproducibility.
AIPerf separates load generation from result processing across processes, using ZMQ coordination to reduce the risk that the benchmark client becomes a server-side performance bottleneck.
Streaming is required for measuring TTFT and ITL because non-streaming responses do not expose first-token or decode-token events.
Poisson arrivals and variable input/output lengths produce wider latency distributions than a fixed single-concurrency workload, reflecting competing requests and differing prefill costs.
The two extra inputs tell the model to emit 128 tokens rather than stopping early. Without them, output length can fall below the target, making throughput lower and results less reproducible.
Streaming exposes the events needed to measure time to first token and inter-token latency. Without streaming, the server sends the completed response at once, so those first- and decode-token measurements are unavailable.
It averages 10 requests per second, varies input and output lengths, and uses a fixed seed for repeatability. The resulting workload has bursts and gaps, and its latency distributions are wider than the fixed baseline.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Anomaly detection identifies meaningful departures from expected behavior. This explainer shows how signals, baselines, context, and alert routing turn unusual measurements into engineer-ready findings.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
