LLM quantization guide: How to do it, and how it helps

Summary

LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.

Takeaways

1. 1

   Weight quantization reduces movement from high-bandwidth memory to on-chip SRAM, benefiting memory-bound token generation; activation quantization improves tensor-core computation during compute-bound prompt processing.

2. 2

   The recommended format depends on the inference engine and hardware: GGUF serves llama.cpp, Ollama, and LM Studio; MLX targets Apple silicon; FP8, NVFP4, AWQ, GPTQ, and compressed-tensors target vLLM and data-center GPUs.

3. 3

   On two H100s serving Llama 3 70B, FP8 weights-and-activations raised throughput from 158 to 474 input tokens/sec and cut time to first token under load from over 30,000ms to about 4,800ms.