These three get compared as though they were competing options. They are not. They solve different problems, and choosing between them is mostly a matter of correctly diagnosing which problem you have.
The one-line version
- RAG gives the model information it does not have, at the moment it needs it
- Fine-tuning changes how the model behaves
- Distillation changes what the model costs to run
Knowledge, behaviour, cost. Almost every real decision resolves cleanly once you work out which of those three is actually failing.
Use RAG when the problem is knowledge
Retrieval-augmented generation fetches relevant material at query time and places it in the context window before the model answers. Reach for it when:
- The model needs facts it was never trained on, such as your internal documents
- The information changes, and stale answers are unacceptable
- You need citations, so a user can verify where an answer came from
- Access differs per user, so the same question should return different material
The last two are decisive and often overlooked. A fine-tuned model cannot cite its sources, because the information is diffused through its weights rather than retrieved from anywhere. And it cannot enforce per-user permissions, because it knows everything it was trained on for everybody.
The most common expensive mistake
Fine-tuning to add knowledge. It is intuitive and it mostly does not work.
Facts injected by fine-tuning are unreliable to retrieve, difficult to update without retraining, prone to being overridden by stronger priors the model already holds, and impossible to attribute. Teams routinely run several fine-tuning cycles trying to get a model to reliably know something a retrieval index would have handled on the first afternoon.
If your failure mode is the model does not know this, the answer is retrieval.
Use fine-tuning when the problem is behaviour
Fine-tuning is right when the model has the capability but not the habits.
- Output format is inconsistent and prompting has stopped improving it
- You need a specific voice, register or house style
- The task is narrow and repeated, and you want reliability rather than range
- You want to shorten prompts by baking instructions into weights
That last one is underrated. If every request carries a thousand tokens of instructions, fine-tuning that behaviour in can cut per-call cost substantially at volume.
Use distillation when the problem is cost
Distillation applies when something already works and you cannot afford to keep running it that way. It does not add knowledge and it does not, on its own, change behaviour. It moves existing capability into a cheaper model. We go into that comparison in more depth in Distillation vs Fine-Tuning.
A working decision sequence
- Does it produce wrong facts, or say it does not know? Retrieval.
- Does it know the answer but present it badly? Prompting first, then fine-tuning if prompting plateaus.
- Is it correct and well-formatted but too slow or expensive? Distillation, or quantisation, or both.
- None of these, and simply not capable enough? None of the three will help. You need a stronger base model or a different decomposition of the task.
That fourth branch is worth taking seriously. A great deal of effort gets spent applying techniques to a capability ceiling that no amount of training or retrieval will raise.
Combining them
A mature production system frequently uses all three:
- Retrieval supplies current, permissioned, citable facts
- Fine-tuning teaches the model to use retrieved context well and answer in your format
- Distillation shrinks the resulting model so serving it is affordable
Built in that order, each step addresses a problem the previous one left standing.
Common questions
Should I use RAG or fine-tuning to add knowledge to a model?
RAG, in almost every case. Fine-tuning is a poor way to add facts: it is expensive to update, it does not reliably override what the model already believes, and it gives you no citation trail. Retrieval puts the information in the context window at query time, where the model can use and attribute it.
Can you use RAG and fine-tuning together?
Yes, and it is a common production pattern. Retrieval supplies the facts while fine-tuning teaches the model how to use retrieved material in your required format and register. They address different failure modes and do not conflict.
Where does distillation fit alongside RAG?
It is orthogonal. Distillation reduces the cost of whatever model you are running. A distilled model can sit behind a retrieval pipeline exactly as its teacher would have, and this is often the cheapest production configuration available.