Loading post…
Red HatSana FayyazBenchmark
Understanding W8A8 INT8 LLM quantization: Accuracy and performance results
Summary
The post evaluates whether W8A8 INT8 quantization can reduce the size of a Llama 3.1 8B Instruct model without sacrificing accuracy or inference performance. Using SmoothQuant to address activation outliers and GPTQ with error compensation for BF16 weights, the model was compressed from 14.9 GB to 8.0 GB after calibration on WikiText-2. Identical zero-shot lm-eval-harness tests across MMLU, ARC Easy, HellaSwag, and IFEval found MMLU, ARC Easy, and HellaSwag deltas smaller than their standard errors, while the IFEval change was described as likely within a similar noise range; the post concludes no measurable degradation. vLLM and GuideLLM tests on one NVIDIA L40S reported 29% more concurrent requests, 24% lower synchronous TTFT, and 44% faster token generation at maximum load, while ITL degraded more quickly under high concurrency because of dynamic activation quantization overhead.
Context
The results compare a BF16 base model with an INT8 W8A8 model using identical hardware and evaluation conditions. The post also contrasts W8A8 with W8A16: W8A8 quantizes weights and activations to enable INT8 tensor-core computation, while W8A16 quantizes only weights and continues computing in BF16.
Approach / What changed
The model used SmoothQuant to modify RMSNorm gamma and reduce activation outliers, then GPTQ to quantize BF16 weights to INT8 with error compensation. WikiText-2 supplied calibration data. Accuracy was measured with lm-eval-harness, while vLLM and GuideLLM generated concurrent workloads with 1,024 input tokens and 512 output tokens on a single NVIDIA L40S GPU.
Takeaways
W8A8 quantization reduced the Llama 3.1 8B Instruct model from 14.9 GB to 8.0 GB, freeing GPU memory for per-request KV caches.
Across MMLU, ARC Easy, HellaSwag, and IFEval, the post reports no measurable accuracy degradation; the main reported deltas were within benchmark variation.
The compressed model supported 44 instead of 34 concurrent requests and produced tokens 44% faster at maximum load, but ITL degraded faster under high concurrency because of dynamic activation quantization overhead.
FAQ
Why did quantization increase the number of concurrent requests?
The compressed model used roughly 7 GB less GPU memory, leaving more memory for the KV cache. Because each active request maintains its own KV cache, the server could increase maximum concurrency from 34 to 44 requests.
Did W8A8 quantization preserve model accuracy?
The reported accuracy differences across MMLU, ARC Easy, HellaSwag, and IFEval were treated as indistinguishable from benchmark variation. The post therefore concludes that quantization did not measurably degrade accuracy on the four tested benchmarks.
What performance trade-off appeared at high concurrency?
ITL degraded faster under high concurrency because of dynamic activation quantization overhead. Even at the maximum tested load, however, the compressed model's latency remained lower than the base model's.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.