---
title: "LLM quantization guide: How to do it, and how it helps"
description: "LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms."
---

# LLM quantization guide: How to do it, and how it helps

[Red Hat](https://yomu.fyi/company/red-hat) · Cedric Clyburn · Sep 2, 2026

**Type:** Tutorial

## Summary

LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.

## Context

Model sizes have roughly doubled every year while GPU memory has not kept pace. The resulting deployment challenge is fitting and serving large open models on realistically available hardware, potentially for many users at once.

## Approach / What changed

The guide explains lower-precision representation for weights and activations, identifies the transformer linear layers where quantization is applied, compares weight and activation benefits, and maps formats to inference engines. It also covers calibrated methods such as GPTQ, AWQ, and SmoothQuant, plus accuracy recovery and serving benchmarks.

## Takeaways

- Weight quantization reduces movement from high-bandwidth memory to on-chip SRAM, benefiting memory-bound token generation; activation quantization improves tensor-core computation during compute-bound prompt processing.
- The recommended format depends on the inference engine and hardware: GGUF serves llama.cpp, Ollama, and LM Studio; MLX targets Apple silicon; FP8, NVFP4, AWQ, GPTQ, and compressed-tensors target vLLM and data-center GPUs.
- On two H100s serving Llama 3 70B, FP8 weights-and-activations raised throughput from 158 to 474 input tokens/sec and cut time to first token under load from over 30,000ms to about 4,800ms.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Quantization](https://yomu.fyi/topic/quantization), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/02/llm-quantization-guide-how-to-do-it--and-how-it-helps)
- Source URL: https://developers.redhat.com/articles/2026/09/02/llm-quantization-guide-how-to-do-it--and-how-it-helps
- Ingested by Yomu: 2026-09-02T04:00:52.258Z

[Read original post](https://developers.redhat.com/articles/2026/09/02/llm-quantization-guide-how-to-do-it--and-how-it-helps)