Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy

Summary

Serving Llama 3.1 8B in BF16 requires roughly 14.9 GB for weights, leaving little VRAM on a 16 GB GPU for KV cache and activations, which limits practical deployment. The workflow benchmarks the base model, applies INT8 W8A8 quantization with llm-compressor, then repeats accuracy and performance tests under identical hardware and load conditions. SmoothQuant modifies weights to reduce activation outliers, while GPTQ uses calibration data and error compensation before rounding weights to INT8; the experiment used 512 WikiText-2 samples and targeted deployment with vLLM. On the Llama 3.1 8B Instruct model, compression reduced storage from 14.9 GB to 8.0 GB, a 46% reduction, and accuracy was preserved across MMLU, ARC, HellaSwag, and IFeval; generation-speed results are reserved for part 2.

Takeaways

1. 1

   W8A8 represents both weights and activations as INT8 during matrix multiplication, allowing use of INT8 tensor cores instead of the BF16 tensor cores used by the uncompressed model.

2. 2

   SmoothQuant and GPTQ are applied sequentially: SmoothQuant addresses activation outliers, while GPTQ calibrates weights and compensates for quantization error before rounding.

3. 3

   The model shrank from 14.9 GB to 8.0 GB. The compressed size exceeds the theoretical 7.5 GB INT8 weight size because some components, including lm_head, remain in BF16 and quantization scales are stored alongside the weights.