All distillation asks a student to copy a teacher. The three families differ in what gets copied, and that choice determines what access you need.
Response-based
The student matches the teacher's output. This is the original formulation and by far the most common, particularly for language models, because it requires nothing but the teacher's answers.
It works across architectures. Teacher and student need share nothing except an output space. It is the only option when the teacher is behind an API, which is why it dominates in practice. See Soft Targets and Dark Knowledge for how the output distribution is actually used.
Feature-based
The student matches the teacher's intermediate activations, not just its final answer. The idea, introduced by FitNets in 2015, is that a network's hidden representations contain structure the output alone does not expose.
This transfers more information and often produces better students, especially thin deep ones. The costs are real: you need access to the teacher's internals, and you need some mapping between layers, which is awkward when architectures differ. Since dimensions rarely match, a learned projection usually sits between them.
Relation-based
The student matches relationships rather than values. Instead of asking that the student's representation of an input resemble the teacher's, you ask that the distances between the student's representations of several inputs resemble the teacher's distances.
This is more flexible, because it does not require the student to occupy the same representational space at all. It only has to preserve the geometry. That makes it attractive when architectures are very different.
Choosing
- API access only: response-based. There is no other option.
- Full access, similar architectures: feature-based is worth trying and usually beats response-based alone.
- Full access, very different architectures: relation-based avoids the correspondence problem.
- In production: combinations are common. Nothing prevents adding a feature-matching term to a response-based objective.
The practical reality is that the 2026 distillation disputes concern response-based methods almost exclusively, because that is all a commercial API exposes. The richer variants require a cooperation the accused parties did not have.
Common questions
What are the three types of knowledge distillation?
Response-based distillation matches the teacher's output distribution. Feature-based distillation matches intermediate activations inside the network. Relation-based distillation matches the relationships between examples or between layers rather than any individual value.
Which type of distillation should I use?
Response-based, unless you have white-box access and a reason to do otherwise. It is the simplest, requires only outputs, and works across different architectures. Feature-based methods can transfer more but require access to the teacher internals and some correspondence between architectures.