The framing that “RAG is for knowledge, fine-tuning is for behaviour” is correct, and it is also where most people stop thinking. The decision that actually matters is not which one to pick. It is which one you need first, because they fail in different ways and cost different amounts to maintain.
What RAG fixes
Retrieval puts relevant text in the prompt at request time. It solves:
- Knowledge that changes. Prices, policies, inventory, documentation. A fine-tune is a snapshot; retrieval is live.
- Attribution. You can show the source. For regulated or customer-facing answers this is often non-negotiable.
- Long-tail coverage. Ten thousand documents nobody wants to train on, retrieved on demand.
- Time to first version. A working RAG prototype takes days. A working fine-tune takes a dataset.
RAG does not fix: an output format the model keeps breaking, a tone you cannot describe, or a task the base model is simply not capable of doing at this size.
What fine-tuning fixes
Fine-tuning changes the weights, so it solves things a prompt cannot reach:
- Rigid output structure. Every response in an exact schema, reliably, without five examples in the prompt.
- Consistent voice. A house style that is easier to demonstrate than to specify.
- Narrow-task accuracy. A 7B model that beats a prompted frontier model on your specific extraction task.
- Latency and cost. Shorter prompts (no examples), smaller model, cheaper inference.
Fine-tuning does not fix: missing facts, changing data, or the need to cite sources.
The cost comparison
| RAG | Fine-tuning | |
|---|---|---|
| Build | 3–10 days | 2–6 weeks including data |
| Main cost | Retrieval infra, embeddings, evals | Labelling, GPU runs, evaluation |
| Recurring cost | Re-indexing, embedding drift | Retraining when the base model or task changes |
| Failure mode | Retrieves the wrong chunk | Learns the wrong pattern, quietly |
| Updating | Re-index a document | Rebuild a dataset, retrain |
The recurring cost is where teams get surprised. A RAG system degrades in visible ways — answers get worse and you can trace why. A fine-tune degrades invisibly: the model still produces fluent output, just slightly wrong, and nothing throws an error.
The framework
Ask in this order:
- Is the problem knowledge or behaviour? Knowledge → RAG. Behaviour → step 2.
- Has a serious prompt been tried? If the best prompt gets you 80% of the way, the last 20% may not be worth a dataset.
- Do you have 500+ examples of the behaviour done correctly? If not, you cannot fine-tune yet. Build the dataset first — and note that building it often teaches you the task well enough to solve it with prompting.
- Will the correct answer change within six months? If yes, retrieval belongs in the loop regardless.
If the answers land on both — they often do — the architecture is a fine-tuned model plus retrieval, not either/or. Fine-tune the behaviour, retrieve the facts.
What people get wrong
Fine-tuning to add knowledge. It works, badly and expensively, and it makes the model confidently wrong about things it half-remembers. Retrieval is better at this in every dimension except latency.
RAG to enforce a format. You can prompt-engineer your way to a valid schema most of the time. “Most of the time” is not a product requirement, and every retry is a token you paid for. This is the case where fine-tuning is clearly correct.
Skipping evaluation. Both approaches need a held-out set and a metric. Without one, you cannot tell an improvement from a regression, and you will ship the worse version because it felt better in three spot checks.
Start with the fine-tuning guide if you have decided to train, or build your dataset first if the answer to question 3 was “not yet”.