Distillation vs fine-tuning

These two get conflated constantly, and there is a good reason for it: in the most common modern setup, distillation is performed by fine-tuning. You collect responses from a large model and fine-tune a smaller one on them. The mechanics are identical. The intent is not.

The short version

Fine-tuning changes what a model does. You take an existing model and train it further on examples from your domain so that it behaves the way you need. The model stays the same size. It costs the same to run afterwards.

Distillation changes what a model costs. You take capability that exists in a large, expensive model and move it into a smaller one, so that you can run it for less. The capability is meant to stay roughly the same. The bill does not.

When you want fine-tuning

Fine-tuning is the right tool when the base model is capable enough but does not behave the way you need. Typical cases:

• It does not follow your output format reliably
• It lacks domain vocabulary, house style, or tone
• It needs to handle a narrow task very consistently
• You have a few hundred to a few thousand good examples of the behaviour you want

With a parameter-efficient method such as LoRA, this is cheap. You are usually looking at hours of compute rather than weeks, and the resulting adapter is small enough to swap in and out.

When you want distillation

Distillation is the right tool when the capability already exists but the economics do not work. Typical cases:

• A large model does the job well but inference is too expensive at your volume
• You need lower latency than a frontier-size model can give you
• You need to run on-premise or on constrained hardware
• You want to own the deployed model rather than depend on an API

The tradeoff is that distillation requires a lot of teacher output. Transferring broad capability rather than one narrow skill needs data volume on a completely different scale from fine-tuning, which is why the industrial-scale extraction campaigns reported during 2026 involved millions of API exchanges rather than thousands.

The comparison, directly

The honest summary is that these questions are not alternatives. They answer different problems, and a lot of production systems end up doing both.

Doing both: the common production pattern

The pattern that shows up repeatedly in practice is:

1. Take a strong general model
2. Fine-tune it on your domain until it does the task well
3. Use that fine-tuned model as a teacher
4. Distill it into something small enough to serve at your volume

You get task-specific behaviour from step two and affordable inference from step four. This is why the two techniques appear together so often, and part of why the vocabulary blurs.

When neither one helps

If the capability you need does not exist in any model you can access, neither technique creates it. Fine-tuning adapts behaviour that is already latent. Distillation compresses capability that already exists. Neither invents reasoning ability that the teacher never had.

This is the most common expensive mistake in the space: teams fine-tune repeatedly against a capability ceiling that fine-tuning cannot raise. If your evaluations are stuck, the question is usually whether you need a better base model, better retrieval, or a different decomposition of the task, not more training runs.

A note on terminology

You will see “distillation” used loosely to mean any training on another model’s outputs, including generating a few thousand synthetic examples to fine-tune on. That usage is now common enough that arguing with it is futile, but it is worth knowing that the technical term originally described something more specific: training a student against a teacher’s full output distribution, which we cover in What Is Knowledge Distillation?

Common questions