Choosing a GPU for fine-tuning is a memory question first and a speed question second. If the card cannot hold the weights, the optimiser state and the activations, nothing else matters. If it can, the difference between cards is hours, and hours have a price you can compare.
The VRAM figures here come from the same accounting the calculator uses: weights, gradients, optimiser state, activations and logits, plus 0.8GB of fixed overhead. They are estimates. The assumptions are stated so you can check them against your own run.
VRAM by model size and method
Rank 16, 2048-token sequences, batch 1, gradient checkpointing on, gradient accumulation doing the work of a larger batch:
| Model | QLoRA (4-bit) | LoRA bf16 | Full fine-tune |
|---|---|---|---|
| Llama 3.2 1B | 1.9 GB | 3.7 GB | 18.6 GB |
| Phi-3.5 mini 3.8B | 3.5 GB | 9.0 GB | 54.7 GB |
| Mistral 7B v0.3 | 5.6 GB | 16.1 GB | 102.9 GB |
| Llama 3.1 8B | 6.2 GB | 17.9 GB | 114.0 GB |
| Gemma 2 9B | 7.2 GB | 20.6 GB | 131.3 GB |
| Qwen2.5 32B | 21.3 GB | 68.8 GB | 461.7 GB |
| Llama 3.1 70B | 44.3 GB | 146.6 GB | 992.1 GB |
The components, so the numbers are not magic. Weights are parameters × 0.55 bytes at NF4 (including quantisation scales) or × 2 bytes at bf16. Trainable parameters are a rank-scaled fraction of the model — 0.2% at rank 16 on most models — and they carry gradients at 2 bytes and AdamW state at 10. Activations scale with batch × sequence length × hidden size × layers, and gradient checkpointing divides that term by four. Logits are batch × sequence × vocabulary, and a chunked loss halves them.
That is why the memory is controllable. Four knobs move it:
| Change | Effect on an 8B QLoRA run |
|---|---|
| Rank 16 → 64 | 6.2 → 6.8 GB |
| Sequence 2048 → 4096, batch 1 → 2, checkpointing on | 6.2 → 8.6 GB |
| Same, checkpointing off | 6.2 → 16.1 GB |
| QLoRA → LoRA bf16 | 6.2 → 17.9 GB |
Gradient checkpointing is the cheapest trade in fine-tuning: it roughly doubles memory headroom for about a third more wall clock. Turn it on unless you have measured that you cannot afford the time.
Consumer vs datacentre cards
The dividing line is 24GB. Below it, consumer cards are the best money in machine learning per token. Above it, you are paying a memory premium and the choices narrow fast.
| Card | VRAM | Dense bf16 TFLOPs | $/hour | $ per 1M tokens |
|---|---|---|---|---|
| RTX 3090 | 24 GB | 71 | 0.22 | 0.138 |
| RTX 4090 | 24 GB | 165 | 0.44 | 0.119 |
| L4 | 24 GB | 121 | 0.44 | 0.162 |
| RTX 6000 Ada | 48 GB | 181 | 0.79 | 0.195 |
| L40S | 48 GB | 362 | 1.19 | 0.147 |
| A100 40GB | 40 GB | 312 | 1.19 | 0.170 |
| A100 80GB | 80 GB | 312 | 1.59 | 0.227 |
| H100 80GB | 80 GB | 989 | 2.69 | 0.121 |
| H200 141GB | 141 GB | 989 | 3.39 | 0.153 |
Read the last two columns together. The 4090 and the H100 are the two most efficient cards per token, at $0.119 and $0.121. The A100 80GB is the least efficient at $0.227, nearly double the 4090 — because you are paying for 80GB of HBM whether your run needs it or not. Datacentre cards are not automatically faster per dollar; they are larger, and they carry features consumer cards lack.
Those features do matter above a certain scale. ECC memory protects long runs from a silent flipped bit. NVLink makes multi-GPU sharding practical. MIG lets a provider slice one card. For a three-hour single-GPU run on 24GB none of it changes your result; for a 70B job across several days, it is the difference between a trustworthy number and a mystery.
One more consumer caveat worth stating plainly: a 24GB card is 24GB. If your run needs 26GB you do not get 26GB, you get an OOM at step 700 after paying for the steps before it.
Renting vs buying
The arithmetic is a single division:
months to break even = card price ÷ (GPU-hours per month × hourly rate)
An illustrative card at $2,000, with the 4090 rate of $0.44/hour:
| GPU-hours per month | Rental cost/month | Break-even |
|---|---|---|
| 50 | $22 | 91 months |
| 100 | $44 | 45 months |
| 400 | $176 | 11 months |
Replace the $2,000 with the actual price you can buy at and the conclusion rarely changes. Below roughly 100 hours a month, renting wins on money and wins again on optionality: you can rent an H100 for the one week you need 80GB and a 4090 for everything else. Owning locks you to one card, adds electricity and cooling, and a card that sits idle still depreciates.
Buying starts to make sense when you are above a few hundred hours a month with a stable configuration, or when the work is sensitive enough that you want the machine on your desk. Even then, compare against the rate for the card you would actually buy, not the cheapest card on the market.
Minimum viable setup
Work backwards from your model and method, then buy or rent one tier above the minimum. The 5% headroom rule matters: the calculator’s “fits” test uses it, and so should you.
| Target | VRAM | Minimum card | Note |
|---|---|---|---|
| 1B QLoRA | 1.9 GB | any 8GB card | Cheap enough that renting is unnecessary |
| 8B QLoRA | 6.2 GB | 8GB works, 12–16GB comfortable | 8.6GB at 4096 tokens, batch 2, checkpointing |
| 8B LoRA bf16 | 17.9 GB | 24GB | The reason to own a 3090 or 4090 |
| 32B QLoRA | 21.3 GB | 24GB is tight, 48GB comfortable | One sequence-length change from an OOM |
| 70B QLoRA | 44.3 GB | 48GB class | RTX 6000 Ada, L40S, or an 80GB A100/H100 |
| 70B LoRA bf16 | 146.6 GB | no single card | Multi-GPU sharding, not priced by this model |
| 8B full fine-tune | 114.0 GB | no single card | Same conclusion as above |
If you are choosing between the 48GB and 80GB tier, the worked comparison is in A100 vs H100: the faster card is often the cheaper run even at nearly double the hourly rate. If you are not buying at all, cheapest GPU cloud prices the same runs by provider, including the setup, storage and egress costs that never appear on a rate card.
Get your own number before you commit: put your model, method, rank, sequence length and batch into the VRAM calculator, and if QLoRA versus bf16 LoRA is still open, the memory and quality comparison is the page that decides it.