A 24GB card is where fine-tuning stops being a memory argument and becomes a scheduling one. At rank 16, 2,048-token sequences, batch 1 and gradient checkpointing, Llama 3.1 8B in QLoRA occupies 6.2GB of the 24, and Qwen2.5 32B occupies 21.3GB — the real ceiling on this tier, with 2.7GB left over for fragmentation and longer sequences. The RTX 4090, the RTX 3090 and the L4 all carry exactly 24GB, so the memory answer is identical on all three. Only the clock and the price per hour differ, and those two decide your bill.
Every number here comes from the same estimate model that drives the VRAM calculator. It is an estimate, not a benchmark: the assumptions are named in each section so you can disagree with them.
Model size ceiling at 24GB
On 24GB, QLoRA reaches Qwen2.5 32B at 21.3GB and stops there; the next model in the line, Llama 3.1 70B, needs 44.3GB and will not fit at any setting worth running. The ceiling is set by the frozen weights plus a fixed per-run overhead, not by the size of the training loop.
The assumptions for the table: NF4 weights at 0.55 bytes per parameter including scales, LoRA rank 16 so 0.2% of parameters are trainable, bf16 gradients, AdamW with fp32 moments and master weights at 10 bytes per trainable parameter, activations at 2 bytes per element for batch 1 × 2,048 tokens × hidden × layers with checkpointing, half-size logits, and 0.8GB of fixed CUDA and allocator overhead.
| Model | Weights | Activations | Total VRAM | Fits 24GB |
|---|---|---|---|---|
| Llama 3.2 3B | 1.8 GB | 0.4 GB | 3.3 GB | yes, with 21GB spare |
| Mistral 7B v0.3 | 4.0 GB | 0.5 GB | 5.6 GB | yes |
| Qwen2.5 7B | 4.2 GB | 0.4 GB | 5.9 GB | yes |
| Llama 3.1 8B | 4.4 GB | 0.5 GB | 6.2 GB | yes |
| Gemma 2 9B | 5.1 GB | 0.6 GB | 7.2 GB | yes |
| gpt-oss-20b (MoE) | 11.6 GB | 0.3 GB | 13.5 GB | yes |
| Qwen2.5 32B | 18.0 GB | 1.3 GB | 21.3 GB | yes, 2.7GB spare |
| Llama 3.1 70B | 38.8 GB | 2.7 GB | 44.3 GB | no |
Qwen2.5 32B is 18.0GB of weights against 21.3GB total, so the last 3.3GB is consumed by things that scale with sequence length and batch — which is why the 32B column is the fragile one. Llama 3.1 70B is not fragile, it is absent: 38.8GB of weights alone exceeds the card by 15GB.
QLoRA is doing the work here. The same models in bf16 LoRA, where weights stay at 2 bytes per parameter: Llama 3.2 3B 7.9GB, Mistral 7B 16.1GB, Llama 3.1 8B 17.9GB, Gemma 2 9B 20.6GB. So bf16 LoRA also fits 24GB, but only to 9B, and at 8B you have 6GB of headroom for a longer sequence instead of 18GB. Full fine-tuning is a different category: Llama 3.2 3B needs 46.4GB, because every parameter carries gradients and optimiser state. 24GB does not do full fine-tuning of anything modern.
QLoRA settings that fit
Four settings move memory, and they are worth ordering by leverage: sequence length and batch first, gradient checkpointing second, rank a distant third. Rank 16 to rank 64 on Llama 3.1 8B adds 0.6GB, because even at rank 64 only 0.8% of parameters are trainable; sequence length doubles the activation and logits terms every time you double it.
| Llama 3.1 8B, QLoRA r16 | Checkpointing on | Checkpointing off |
|---|---|---|
| 512 tokens | 5.6 GB | 6.1 GB |
| 1,024 tokens | 5.8 GB | 6.7 GB |
| 2,048 tokens | 6.2 GB | 8.1 GB |
| 4,096 tokens | 7.0 GB | 10.8 GB |
| 8,192 tokens | 8.6 GB | 16.1 GB |
At 8B the card is not the constraint at any of these lengths, which is the point: spend the spare memory on longer sequences instead of a bigger model. Batch behaves the same way — 6.2GB at batch 1, 7.0GB at 2, 8.6GB at 4, 11.8GB at 8 — and batch is the wrong thing to buy, because the same tokens per step come cheaper as gradient accumulation, which holds one microbatch of activations.
For Qwen2.5 32B the arithmetic is tight and sequence length decides it. Rank 16 at 2,048 tokens is 21.3GB and at 3,072 tokens it is 22.1GB. At 4,096 tokens it is 22.9GB, which clears 24GB in raw terms but not the 5% headroom rule the calculator uses: 24 × 0.95 is 22.86. A card that reports 21.3GB used and then dies at step 400 is usually not out of weights, it is out of contiguous blocks, so the 32B configuration that survives is short-sequence and conservative.
The settings that make 32B work on 24GB: rank 16, 2,048-token sequences, batch 1 with gradient accumulation of 8 or more, gradient checkpointing, NF4 with double quantisation and bf16 compute, paged AdamW, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True against fragmentation. Those are the defaults the config generator writes for a 24GB card, and the reasoning behind each flag is on the QLoRA page.
Throughput you can expect
On these three cards an 8B QLoRA run should land near 442 tokens per second on a 3090, 1,027 on a 4090 and 753 on an L4, if the pipeline can keep the GPU fed. Those are compute-bound estimates: efficiency × dense bf16 TFLOPs ÷ (6 × parameters), with efficiency at 0.3 for QLoRA to account for dequantisation.
| Model (QLoRA r16, 2,048 tokens, batch 1) | RTX 3090 | RTX 4090 | L4 |
|---|---|---|---|
| Llama 3.1 8B | 442 tok/s | 1,027 tok/s | 753 tok/s |
| Gemma 2 9B | 384 tok/s | 893 tok/s | 655 tok/s |
| Qwen2.5 32B | 108 tok/s | 252 tok/s | 184 tok/s |
Treat each of those as a ceiling rather than a prediction. The model uses dense bf16 throughput, so it gives no credit for faster attention kernels and no penalty for a dataloader that stalls; eval passes, checkpoint writes and shuffling all come out of the same wall clock. Time 50 steps on your own job and multiply.
Cost follows from throughput, and the ranking is not the ranking of the hourly rates. The 3090 is the cheapest card per hour at a median $0.22 and the most expensive of the three per token, because it is 2.3× slower than the 4090.
| Card | Median $/hour | 8B QLoRA $ per 1M training tokens | 22.5M-token run: hours | Cost |
|---|---|---|---|---|
| RTX 3090 24GB | 0.22 | 0.138 | 14.1 | $3.11 |
| RTX 4090 24GB | 0.44 | 0.119 | 6.1 | $2.68 |
| L4 24GB | 0.44 | 0.162 | 8.3 | $3.65 |
A 22.5M-token run is 5,000 examples at 1,500 tokens over 3 epochs, a realistic first instruction-tuning job. Same work, same result, 2.3× the wall clock and 16% more money on the 3090. Rates are marketplace medians from the estimate model, not quotes; check the provider before you commit. The GPU comparison covers the rest of the field.
Upgrade paths
Leave 24GB for one of three reasons: you want 70B, you want 32B at 4,096-token sequences or longer, or you want bf16 LoRA above 9B. All three are memory problems, and all three are solved at 48GB, which is the next real step rather than 40GB.
| Card | VRAM | Median $/hour | 70B QLoRA (44.3GB) | 8B QLoRA tok/s |
|---|---|---|---|---|
| RTX 6000 Ada | 48 GB | 0.79 | fits | 1,127 |
| L40S | 48 GB | 1.19 | fits | 2,254 |
| A100 40GB | 40 GB | 1.19 | does not fit | 1,943 |
| A100 80GB | 80 GB | 1.59 | fits | 1,943 |
| H100 80GB | 80 GB | 2.69 | fits | 6,158 |
The A100 40GB is the trap: 2.7× the hourly rate of a 4090 with the same ceiling on a 70B job. 40GB is a strange size once QLoRA exists, because 32B needs 21.3GB and 70B needs 44.3GB, and nothing common lands in between.
The other reason to upgrade is time rather than money. An 8B QLoRA run costs $0.121 per million training tokens on an H100 and $0.119 on a 4090, because the H100 is 6× faster for 6.1× the hourly rate. If the run has to finish before a meeting, that is the whole argument; if it can run overnight, it buys you nothing.
Work the numbers for your own model, sequence length and epochs in the cost calculator: a 24GB card covers more configurations than the model-size table suggests once you trade sequence length for parameters, and the cheapest upgrade is often a shorter sequence rather than a bigger card.