Software Development

Knowledge Distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

Publisher / source

Orchestra-Research/AI-Research-SKILLs (found on skills.sh, an awesome list)

Upstream record

Orchestra-Research/AI-Research-SKILLs

Version

Version 1

Published

2026-11-01

Publication review

Approved for publication

Content fingerprint

bc27086dc02840cebef4db34db4b8406b37ff349d6e2a52848646c8daf24ff3e