Soft targets and dark knowledge

The core claim of distillation is that a teacher's wrong answers are more useful than its right ones. That sounds like a paradox and is not.

Where the information actually is

Consider a model classifying an image of a dog. A hard label says dog = 1 and everything else 0. A trained model says something closer to dog = 0.92, wolf = 0.06, cat = 0.019, car = 0.0000004.

The hard label tells the student one thing: this is a dog. The teacher's distribution tells it considerably more. Dogs resemble wolves closely. They resemble cats somewhat. They resemble cars essentially not at all. That is real structure about the world, learned at great expense, and it is present in every training example rather than being something the student must rediscover.

Geoffrey Hinton called this dark knowledge: information the model clearly possesses, expressed nowhere in its top answer.

Why hard labels waste most of the signal

A one-hot label over a thousand classes carries very little information per example. It identifies one class and says nothing about the relationship between the other 999.

A full distribution over those same classes carries far more, and crucially it carries information the student can use to constrain its own representation. This is the main reason distilled models often train faster and on less data than models trained from scratch on hard labels.

The problem temperature solves

There is a practical obstacle. A well-trained teacher is confident, and confident distributions are extremely peaked:

dog  0.9994
wolf 0.0004
cat  0.0002
car  0.00000003

The structure is still there, but numerically it has almost no influence. Those differences contribute nearly nothing to the gradient, and the student learns roughly what a hard label would have taught it.

How temperature works

Temperature scaling divides the logits by a constant T before the softmax:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

At T = 1 you get the ordinary output. As T rises the distribution flattens and the small probabilities rise into a range where they matter. The same four values at T = 5 might read closer to 0.62, 0.21, 0.14, 0.03. The ordering is preserved. The structure is now visible to the optimiser.

Push T too high and you flatten toward uniform, destroying the signal you were trying to expose. Typical working values are 2 to 10, with more confident teachers wanting higher values.

The combined loss

The student is usually trained against both the teacher and the ground truth:

L = alpha * T^2 * KL(student_T || teacher_T)  +  (1 - alpha) * CE(student, labels)

The first term is the soft-target loss, measured as KL divergence between the two temperature-softened distributions. The second is ordinary cross-entropy against the real labels, which keeps the student anchored when the teacher is wrong. alpha balances them, and higher values trust the teacher more, which suits a strong teacher and scarce or noisy labels.

Why T squared

This factor is the detail most implementations get wrong at least once.

Gradients flowing through a softmax at temperature T scale as roughly 1/T^2. Without correction, raising the temperature to expose more structure would simultaneously shrink that term's contribution, so changing T would silently change the effective alpha as well. Multiplying by T^2 cancels it, and the two hyperparameters become independent, which is the only way tuning them is tractable.

Note also that temperature applies only during training. At inference the student runs at T = 1 like any other model.

What happens to all this with language models

Everything above assumes access to the teacher's full output distribution. Most modern LLM distillation does not have it, because commercial APIs return sampled text rather than logits.

In that setting the elegant machinery is unavailable. You are training on observed outputs, which is closer to ordinary supervised fine-tuning on machine-generated data, and the dark knowledge is present only implicitly in which continuations the teacher happened to produce. Sampling many responses per prompt at raised sampling temperature partially recovers distributional information, but it is a coarse substitute.

This is exactly why providers restrict logprob access. Full distributions make genuine white-box distillation dramatically more efficient, and withholding them is one of the cheapest defences available. See How to Tell If a Model Was Distilled for what that costs on the detection side.

Common questions