---
title: "Making Knowledge Distillation Cheap Enough to Run at Scale"
description: "Knowledge distillation trains a smaller student to match a larger teacher's capabilities, but conventional online training keeps both models loaded and materializes full vocabulary distributions at every step, making long-context recovery expensive. The proposed system caches the teacher's top-100 logits once and introduces a fused, chunked KL-divergence loss that avoids storing the full vocabulary-by-sequence grid or the student's full logits. In benchmarks with an 8K-token context on one H200, offline fused chunked KL used 58.3GB peak memory versus 102.8GB for online training, while all four methods reached near-identical training loss. At 32K tokens, the fully chunked loss reduced isolated kernel memory from 85.2 GiB to 5.45 GiB; distilling GPT-OSS 20B at that context cut step time from 57.0 to 12.23 seconds and reduced the setup from four GPU nodes to one."
---

# Making Knowledge Distillation Cheap Enough to Run at Scale

[Hugging Face](https://yomu.fyi/company/hugging-face) · Antonio Tiene, Iker García-Ferrero, Ali Hashemi, Bakbergen Ryskulov · Aug 10, 2026

**Type:** Problem & solution

## Summary

Knowledge distillation trains a smaller student to match a larger teacher's capabilities, but conventional online training keeps both models loaded and materializes full vocabulary distributions at every step, making long-context recovery expensive. The proposed system caches the teacher's top-100 logits once and introduces a fused, chunked KL-divergence loss that avoids storing the full vocabulary-by-sequence grid or the student's full logits. In benchmarks with an 8K-token context on one H200, offline fused chunked KL used 58.3GB peak memory versus 102.8GB for online training, while all four methods reached near-identical training loss. At 32K tokens, the fully chunked loss reduced isolated kernel memory from 85.2 GiB to 5.45 GiB; distilling GPT-OSS 20B at that context cut step time from 57.0 to 12.23 seconds and reduced the setup from four GPU nodes to one.

## Context

Online knowledge distillation is expensive because the teacher and student must remain loaded together, the teacher is recomputed at every step, and full vocabulary-by-sequence probability tensors consume substantial VRAM. These costs make long-context recovery and large-scale experimentation difficult.

## Approach / What changed

The system performs offline distillation by caching the teacher's top-100 logits per token, then trains against that reusable cache. Its fused, chunked KL loss keeps the teacher sparse, processes sequence slices incrementally, fuses the student's output projection into loss computation, discards each chunk, and recomputes chunks during backpropagation instead of storing the full logits grid.

## Takeaways

- Caching the teacher's top-100 logits removes the teacher from student training and allows one cache to be reused across multiple ablations.
- Fused chunked KL avoids materializing the student's full vocabulary-by-sequence logits grid, reducing memory growth with sequence length; it trades this reduction for an extra output projection during backward computation.
- At 32K tokens, fully chunked loss lowered isolated loss-kernel memory from 85.2 GiB to 5.45 GiB, while GPT-OSS 20B distillation fell from four GPU nodes to one and improved step time about fivefold.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation)
- Source URL: https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation
- Ingested by Yomu: 2026-08-29T05:40:56.607Z

[Read original post](https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation)