Teacher and student models

The teacher and student framing makes distillation sound simple: pick a good model, pick a small model, train the second on the first. In practice the choice of teacher is where most of the difficulty lives, and the intuitive answer is frequently wrong.

The capacity gap

The most counter-intuitive result in this area is that a better teacher can produce a worse student.

If the gap in capacity between the two models is large enough, the student physically cannot represent the function the teacher has learned. Asked to match a target it has no capacity to fit, it approximates badly across the board rather than learning well the parts it could have handled. Distilling from a mid-sized teacher often produces a better small model than distilling from the largest one available.

The documented fix is teacher assistant distillation: insert an intermediate model between the two. The large teacher distils into the assistant, the assistant distils into the student. Each hop crosses a gap small enough to be learnable. It costs an extra training run and is usually worth it.

Multiple teachers

Nothing requires exactly one teacher. Ensemble distillation trains the student against the aggregate output of several models, and this generally produces better generalisation than any individual teacher, because idiosyncratic errors average out while genuine structure reinforces.

This was in the technique from the beginning. The 2015 paper that named distillation was substantially about compressing an ensemble into a single deployable model.

It has a side effect that became relevant in 2026: a student trained against several teachers is much harder to attribute to any one of them, because no single teacher's characteristic behaviour survives at recoverable strength. We cover the consequences in How to Tell If a Model Was Distilled.

What transfers, and what does not

Not all capability moves equally well.

• Transfers well: pattern recognition, formatting, domain fluency, and factual recall in areas the teacher covered densely
• Transfers unevenly: multi-step reasoning, which often needs the teacher's intermediate steps rather than just its answers
• Transfers poorly: safety behaviour and refusal calibration

That last point is consistent and important. Capability distils readily; alignment does not. A student trained on a well-aligned teacher's outputs typically underperforms it on refusal benchmarks, because refusals are a small fraction of any sampled corpus and the behaviour is easy to wash out. Anyone distilling a model and assuming they inherited its safety properties has assumed wrong.

Choosing a student architecture

The student does not need to resemble the teacher. In response-based distillation the only requirement is that both produce outputs over the same space. Everything else is free.

In practice the choice is driven by deployment. Latency target, memory budget and hardware determine size, and size determines how much capability you can realistically expect to retain. It is worth setting that expectation before training rather than after.

How much data

The volume required scales with the breadth of what you are transferring. A single narrow skill can move with thousands of examples. General capability across many domains requires orders of magnitude more, because the student needs coverage of the whole input distribution rather than one slice of it.

This is why the extraction campaigns described during 2026 involved tens of millions of exchanges. Transferring broad frontier capability is a data problem before it is a training problem.

The practical sequence

1. Fix the student size from your deployment constraints, not from ambition
2. Pick a teacher no more than roughly an order of magnitude larger, or plan an assistant hop
3. Decide what you actually need transferred, and build prompts that cover that distribution
4. Evaluate on the capabilities you care about, and separately on safety, because the second will regress

Common questions