---
title: Knowledge Distillation — AI Glossary
description: Knowledge distillation trains a smaller "student" model to reproduce the behavior of a larger "teacher" model, learning from the teacher's outputs rather than only from labels. The result is a compact model that retains much of the teacher's capability while being cheaper and faster to run.
canonical: https://prompeteer.ai/glossary/distillation
---

# Knowledge Distillation

Knowledge distillation trains a smaller "student" model to reproduce the behavior of a larger "teacher" model, learning from the teacher's outputs rather than only from labels. The result is a compact model that retains much of the teacher's capability while being cheaper and faster to run.

## Definition

Introduced by Hinton and colleagues, distillation transfers the "dark knowledge" in a teacher's full probability distribution — not just its top answer — to a smaller student. The student is trained to match these soft targets, which carry richer information than hard labels and help it generalize.

Distillation is a key tool for deploying capable models under tight latency and cost budgets, and for creating small on-device models. It is often combined with quantization and pruning, and modern variants distill from large language models to produce efficient specialized ones.

## Sources

- [Hinton et al. (2015), Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)
- [Sanh et al. (2019), DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108)

## Related terms

- [fine-tuning](https://prompeteer.ai/glossary/fine-tuning)
- [quantization](https://prompeteer.ai/glossary/quantization)
- [lora](https://prompeteer.ai/glossary/lora)
- [inference](https://prompeteer.ai/glossary/inference)