Fine-tuning changes how a model behaves. It does not reliably change what the model knows, and it will not fix a model that was never given the right information in the first place. Most failed fine-tuning projects fail on that distinction before a single GPU hour is spent.
This guide covers the decision, the method, the data, the cost and the evaluation, in the order you will actually hit them.
When fine-tuning is the right tool
Fine-tuning earns its cost in four situations:
- Format and structure. You need every output in a rigid shape — a JSON schema, a house style, a fixed clinical phrasing — and prompting keeps drifting.
- Tone and persona. The behaviour is a style that is hard to describe but easy to demonstrate with a few hundred examples.
- Task specialisation. A narrow classification, extraction or rewriting task where a 7B model with the right examples beats a 70B model with a generic prompt, at a fraction of the inference cost.
- Latency and unit economics. A fine-tuned 8B model can replace a prompted frontier model on a narrow task, and inference gets cheaper by an order of magnitude.
It does not earn its cost when the problem is missing knowledge that changes weekly, when you have fewer than a few hundred good examples, or when nobody has tried a serious prompt first.
The order of operations is: prompt engineering, then retrieval, then fine-tuning. Skipping straight to training is the most expensive way to discover that your prompt was the problem.
The full decision framework, with cost comparisons, is in fine-tuning vs RAG.
Choosing a method
| Method | Trainable parameters | VRAM for an 8B model | When it wins |
|---|---|---|---|
| Full fine-tune | 100% | ~112GB+ | New domain, large dataset, you own the hardware |
| LoRA (bf16) | ~0.2% at rank 16 | ~20GB | Quality matters and you have a 24GB+ card |
| QLoRA (4-bit) | ~0.2% at rank 16 | ~6GB | The default. Fits a 7–8B model on consumer hardware |
| DPO / preference tuning | ~0.2% | Same as LoRA | You have preference pairs and need alignment, not capability |
Start with QLoRA. It gets you a working model on a single 24GB card for a few dollars, and the quality gap to LoRA at bf16 is small on most narrow tasks — measurable, but usually smaller than the gap caused by bad data.
If you want the mechanics, read how QLoRA works and LoRA vs QLoRA.
Preparing the data
This is where 80% of the outcome is decided. Three rules:
Format matches training objective. For instruction tuning, one example is one instruction-response pair. For chat models, apply the model’s chat template — not a generic one. A Llama 3.1 model trained on Mistral’s template learns the wrong control tokens and produces mangled output.
{"messages": [
{"role": "user", "content": "Extract the invoice total from: ..."},
{"role": "assistant", "content": "{\"total\": 1284.50, \"currency\": \"EUR\"}"}
]}Quality beats volume. 500 consistent, correct examples outperform 5,000 noisy ones. Deduplicate aggressively — near-duplicates teach the model to memorise rather than generalise.
Hold out a real test set. 5–10% stratified by the thing you care about, not a random slice. If your task has five output categories and one of them appears twice in the test set, you cannot measure it.
Dataset construction gets its own treatment in building a fine-tuning dataset. Sizing guidance in short: below ~100 examples you are better off with few-shot prompting; 500–5,000 is the useful band for most narrow tasks; beyond ~20,000 you should ask whether a better base model solves it more cheaply.
Estimating the cost before you rent
Two numbers decide your bill: how much VRAM you need, and how many GPU hours the run takes.
VRAM is roughly:
weights = params × bytes_per_param (2 bf16 · 0.55 NF4)
gradients = trainable_params × 2
optimiser = trainable_params × ~10 (AdamW fp32 m, v, master copy)
activations = batch × seq × hidden × layers × 2 × ckpt_factor
logits = batch × seq × vocab × 2 × ce_factor
overhead ≈ 0.8 GB (CUDA context, fragmentation)A worked example — Llama 3.1 8B, QLoRA rank 16, 2048 tokens, batch 1, gradient checkpointing on:
| Component | GB |
|---|---|
| Weights (4-bit) | 4.42 |
| Gradients | 0.03 |
| Optimiser | 0.16 |
| Activations | 0.54 |
| Logits | 0.26 |
| Overhead | 0.80 |
| Total | 6.21 |
That fits a 24GB card with room for batch 2 or a longer sequence. The VRAM calculator does this for any model and card, and shows which term dominates as you scale.
GPU hours come from tokens, not examples:
tokens = examples × tokens_per_example × epochs
hours = tokens / (throughput_tokens_per_second × 3600)On a 4090, an 8B QLoRA run at 2048 tokens processes roughly 1,000–1,500 tokens/second depending on the framework. 5,000 examples × 800 tokens × 3 epochs is 12M tokens, so about 2.2–3.3 hours — under $1.50 at $0.44/hour.
The intuition to keep: training a small model is cheap; getting the data and evaluation right is the expensive part.
Running it
A minimal QLoRA run with TRL and PEFT. The parameters below are a defensible starting point, not a universal answer:
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
peft = LoraConfig(
r=16,
lora_alpha=32, # alpha = 2r is a safe default
lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
args = SFTConfig(
per_device_train_batch_size=1,
gradient_accumulation_steps=8, # effective batch 8
learning_rate=2e-4, # LoRA tolerates ~10x the full-FT rate
num_train_epochs=3,
max_seq_length=2048,
gradient_checkpointing=True,
bf16=True,
logging_steps=10,
save_strategy="epoch",
)Three settings cause most of the pain:
- Learning rate. LoRA trains a small number of parameters and needs a much higher rate than full fine-tuning.
2e-4is a normal starting point;2e-5will look like nothing is happening. - Gradient accumulation. Batch size 1 with accumulation 8 gives an effective batch of 8 without the memory cost. If loss is noisy, raise accumulation before you raise the base batch.
- Epochs. One to three. More epochs on a small dataset means memorisation, and the loss curve will look great while the model gets worse.
Evaluating it
The evaluation that matters compares your fine-tune against the same base model with a good prompt — not against nothing.
Three layers, cheapest first:
- Task metric. Exact match, F1, schema validity — whatever your task actually produces. Run it on the held-out set.
- Regression check. A sample of general prompts the model used to handle. Fine-tuning degrades unrelated ability, and it does so quietly.
- Human spot check. 20–30 outputs, read by someone who knows what good looks like. This catches the failures metrics miss: right format, wrong content.
A fine-tune that improves the task metric by 12 points and drops general instruction-following by 20 is usually a net loss. Evaluating a fine-tuned model properly covers the failure modes, including the benchmark traps that make people believe a regression is an improvement.
What it costs, end to end
| Item | Typical |
|---|---|
| 8B QLoRA run, 5k examples, 3 epochs | 2–4 GPU hours |
| At marketplace 4090 prices ($0.44/h) | $1–2 |
| At A100 80GB prices ($1.59/h) | $6–12 |
| Data labelling, 1k examples at $0.50 | $500 |
| Your time on evaluation | The real cost |
The GPU rental is not the budget line that decides whether this project succeeds. The data and the evaluation are.
The short version
- Fine-tune for behaviour, retrieve for knowledge.
- QLoRA on a rented 24GB card is the cheapest way to find out whether fine-tuning helps at all.
- 500 good examples beat 5,000 noisy ones.
- Compare against the prompted base model, or you are measuring nothing.
- Plan the VRAM before you rent. It takes two minutes and saves a failed run.