QLoRA is the reason a 7B model can be fine-tuned on a single consumer graphics card. It is a combination of three techniques, and knowing which one does the work tells you where the trade-offs land.
What QLoRA actually does
The base model’s weights are frozen and stored in 4-bit NormalFloat, while small LoRA adapters train in 16-bit on top. During the forward and backward pass, blocks of weights are dequantised on the fly, used, and discarded.
Three parts:
- NF4 quantisation. A 4-bit data type whose quantisation levels are placed at the quantiles of a normal distribution. Neural network weights are roughly normally distributed around zero, so NF4 spends its 16 levels where the weights actually are, instead of evenly across the range.
- Double quantisation. The quantisation constants themselves are quantised, saving about 0.37 bits per parameter — roughly 3GB on a 65B model.
- Paged optimisers. Optimiser state can spill to CPU memory when the GPU runs short, instead of throwing an out-of-memory error. It buys survival at the cost of speed.
The original paper trained a 65B model on a single 48GB GPU while matching 16-bit LoRA quality on their benchmarks. That result is why QLoRA became the default.
The memory maths, in numbers
Bytes per parameter, by method:
| Method | Weights | Gradients | Optimiser | Total per param |
|---|---|---|---|---|
| Full, bf16 | 2 | 2 | ~12 | ~16 |
| LoRA, bf16 | 2 | trains ~0.2% | trains ~0.2% | ~2.0 |
| QLoRA, NF4 | 0.55 | trains ~0.2% | trains ~0.2% | ~0.56 |
For Llama 3.1 8B at rank 16, 2048 tokens, batch 1, with gradient checkpointing:
| Component | QLoRA | LoRA bf16 |
|---|---|---|
| Weights | 4.42 GB | 16.06 GB |
| Gradients | 0.03 GB | 0.10 GB |
| Optimiser | 0.16 GB | 0.64 GB |
| Activations | 0.54 GB | 0.54 GB |
| Logits | 0.26 GB | 0.26 GB |
| Overhead | 0.80 GB | 0.80 GB |
| Total | 6.21 GB | 18.40 GB |
The whole difference is the weights — 11.6GB of it. Everything else is the same, because the trainable surface is identical. Run your own numbers in the VRAM calculator.
What it costs you
Quality. Slight. On most narrow tasks the gap between QLoRA and bf16 LoRA is under a point on task metrics, and it is swamped by data quality. On tasks that need fine-grained precision — arithmetic, strict schema adherence, exact copying — 4-bit weights do degrade output, and bf16 LoRA is worth the extra memory.
Speed. QLoRA is slower per step than bf16 LoRA because every weight is dequantised before use. Expect 20–40% lower throughput. On a rented 4090 that is cents; on a 70B run it is hours.
Merging. You can merge LoRA adapters into the base weights. You cannot cleanly merge into 4-bit weights — you merge into the bf16 base, then re-quantise, and lose a little fidelity in the round trip. If you plan to serve the merged model, train in bf16 LoRA and avoid the second quantisation.
When to use QLoRA, and when not to
Use QLoRA when:
- The model does not fit your GPU at bf16. This is the common case below 24GB.
- You are experimenting and want the cheapest possible iteration loop.
- You are fine-tuning 30B+ on a single 80GB card.
Use bf16 LoRA when:
- The model fits comfortably and you care about per-step throughput.
- You need maximum fidelity on a precision-sensitive task.
- You intend to merge and serve the weights, so you want to skip the re-quantisation round trip.
Use full fine-tuning when:
- The change is a genuine domain shift with tens of thousands of examples, and you have the memory budget. Most teams do not need this.
A starting configuration
from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype="bfloat16",
)compute_dtype matters: keep computation in bf16 even though storage is 4-bit. Computing in fp16 on a model with large activations is a reliable route to NaN loss.
Pair it with gradient_checkpointing=True, rank 16, alpha 32, and a learning rate around 2e-4. Then measure before you scale the run up — see LoRA vs QLoRA for the decision table by GPU, and LoRA fine-tuning for how rank and alpha actually behave.