How much VRAM does your fine-tune need?

How the estimate works

Every term is computed from the model's architecture and your settings:

weights      = params x bytes_per_param   (2 bf16 · 0.55 NF4)
gradients    = trainable x 2
optimiser    = trainable x 10              (AdamW fp32 m, v, master)
activations  = batch x seq x hidden x layers x 2 x ckpt_factor
logits       = batch x seq x vocab x 2 x ce_factor
overhead     = 0.8 GB

trainable is the LoRA adapters only, scaled by rank, for LoRA and QLoRA; it is every parameter for a full fine-tune.

What the estimate assumes

• Gradient checkpointing is on, which cuts activations roughly fourfold.
• Mixed precision bf16 compute, with AdamW in fp32.
• Sequence packing is off. Packing raises throughput but can raise peak memory.
• No ZeRO sharding, no CPU offload, single GPU.

Expect real runs to land within roughly 10-25% of these numbers. The gap is usually framework overhead and allocation fragmentation, not the formula.

Common reasons you still OOM

• Sequence length is not what you think. One long example in a batch sets the peak for the whole batch.
• Evaluation runs in the same process. Loading a second model or a full eval set doubles you over.
• Wrong compute dtype. fp16 on a large model produces NaN loss before it produces an OOM, which often reads as a memory bug.
• Merging after training. Merging needs the bf16 base plus the adapter, so the peak sits at the end, not the start.