Knowledge distillation is a technique for transferring the capabilities of a large, expensive model (the teacher) into a smaller, cheaper one (the student). The student is trained not to reproduce the correct answers, but to reproduce the teacher’s reasoning about the answers.
The core idea
Suppose you train an image classifier and show it a photo of a dog. A
conventional training label says: dog = 1, cat = 0, car = 0. That is a
hard label, one right answer, everything else equally wrong.
But a well-trained teacher does not output that. It outputs something like
dog = 0.92, cat = 0.07, car = 0.00001. Look at what that says: the
teacher believes this image is slightly cat-like and
overwhelmingly not car-like. The relationship between the wrong answers
encodes something real about how the world is structured, cats and dogs
genuinely do resemble each other more than either resembles a car.
Geoffrey Hinton called this dark knowledge: the information hiding in the wrong answers. Hard labels throw it away. Distillation keeps it.
Why it works so well
Training a student on the teacher’s full probability distribution, its soft targets, turns out to be dramatically more efficient than training on hard labels:
- More information per example. A hard label carries roughly one bit of signal. A soft distribution over thousands of classes carries far more, so the student learns more from each training example.
- A smoother optimization target. Soft targets act as a regularizer, giving the student a gentler surface to descend and reducing overfitting.
- Inherited generalization. The student learns the teacher’s mistakes and uncertainties too. That sounds bad, but it means the student inherits hard-won structure the teacher discovered over an enormously expensive training run.
The result is that a student can often reach a large fraction of the teacher’s performance at a small fraction of its size and inference cost.
The temperature knob
There is a practical problem: a confident teacher outputs distributions like
0.99, 0.008, 0.0000001. The interesting structure is squashed almost
to zero.
The fix is temperature scaling. The softmax is divided by a temperature
parameter T before normalizing. At T = 1 you get the normal
output; at T = 4 or higher the distribution flattens and the relative
structure among the low-probability classes becomes visible and trainable.
The student is then trained on a combined objective, part matching the teacher’s softened distribution, part matching the true hard labels:
L = α · T² · KL(student_soft ‖ teacher_soft) + (1 − α) · CE(student, hard_labels)
The T² factor is a technical necessity: gradients through the softened
softmax scale as 1/T², so without it the soft-target term would shrink
as you raise the temperature.
A short history
- 2006, Buciluă, Caruana and Niculescu-Mizil publish Model Compression, showing a small network can be trained to mimic a large ensemble.
- 2015, Hinton, Vinyals and Dean publish Distilling the Knowledge in a Neural Network, introducing temperature scaling and the term “distillation.” This is the paper everyone cites.
- 2015, FitNets extends the idea from outputs to intermediate representations: the student matches the teacher’s internal features, not just its final answer.
- 2016, Kim and Rush adapt distillation to sequence generation, making it applicable to translation and, eventually, language models.
- 2018, Born-Again Networks shows something strange: a student with the same architecture as its teacher often outperforms it. Distillation is not only compression.
What changed with large language models
Classic distillation assumed you could see inside the teacher, its logits, its activations. Most modern LLM distillation cannot. You have an API that returns text.
So the technique adapted. Black-box distillation simply collects a large set of prompts, records the teacher’s text responses, and fine-tunes the student on those pairs. No logits, no internals, just imitation of observed behavior. Stanford’s Alpaca (2023) trained on roughly 52,000 instruction-following examples generated by an OpenAI model; Vicuna trained on shared ChatGPT conversations. Both got remarkably far on remarkably little compute.
This is where distillation stopped being a purely technical topic. Compressing your own model is uncontroversial engineering. Training a competitor on someone else’s model outputs raises questions about terms of service, intellectual property, and, once national AI capability entered the conversation, export control and sovereignty.
Those questions are the subject of Track 5. The technique itself is over twenty years old and entirely mundane. What changed is who owns the teacher.