Knowledge Distillation

Definition

Introduced by Hinton and colleagues, distillation transfers the "dark knowledge" in a teacher's full probability distribution — not just its top answer — to a smaller student. The student is trained to match these soft targets, which carry richer information than hard labels and help it generalize.

Distillation is a key tool for deploying capable models under tight latency and cost budgets, and for creating small on-device models. It is often combined with quantization and pruning, and modern variants distill from large language models to produce efficient specialized ones.

Frequently asked questions

Sources

• Hinton et al. (2015), Distilling the Knowledge in a Neural Network
• Sanh et al. (2019), DistilBERT, a distilled version of BERT

Related terms

• Fine-Tuning
• Quantization
• LoRA (Low-Rank Adaptation)
• Inference