Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

Summary

Structural compression and 4-bit quantization can reduce an LLM’s reasoning, mathematical, and coding capabilities, while conventional healing methods either require costly quantization-aware training or distill from an already degraded checkpoint. Quantization-Aware Healing (QAH) instead distills the original, full-size, full-precision model into a structurally compressed student running in MXFP4, using KL divergence over teacher logits. A chunked KL loss allows healing on documents up to 32k tokens within a fixed GPU memory budget. Applied to GPT-OSS 120B compressed to 60B parameters, QAH’s 4-bit model outperformed its 60B bfloat16 counterpart on 7 of 9 benchmarks, including gains of 7.4 points on AA-LCR and 5.6 on AIME 2025. In a separate comparison, QAH reached its peak about seven times faster than QAT and avoided QAT’s later 19-point collapse.

Takeaways

1. 1

   A GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4 beat its 60B bfloat16 counterpart on 7 of 9 benchmarks.

2. 2

   QAH reached an average benchmark peak of 54.9 in about 100 steps, compared with 54.6 for QAT at about 700 steps; QAT later lost nearly 19 points by step 1,200.

3. 3

   The QAH model uses roughly four times less weight memory than the bfloat16 student and surpassed the full-size teacher on LiveCodeBench, scoring 66.5 versus 66.0.