The A100 and the H100 differ in one thing that matters for fine-tuning: how fast they turn a token into an update. Memory is identical for a given configuration, because memory is a property of your model, method and batch, not of the card. Once you accept that, the choice comes down to price per hour multiplied by hours per run, and the answer is not the one the hourly rates suggest.
Every number below comes from the same estimate model that drives the calculator. It is an estimate: the assumptions are named, and you should argue with them rather than trust them.
Throughput and memory
Memory first, because it decides whether a card is even a candidate. For rank 16, 2048 tokens, batch 1, gradient checkpointing on:
| Model and method | VRAM needed | A100 40GB | A100 80GB | H100 80GB |
|---|---|---|---|---|
| Llama 3.1 8B, QLoRA | 6.2 GB | fits | fits | fits |
| Llama 3.1 8B, LoRA bf16 | 17.9 GB | fits | fits | fits |
| Qwen2.5 32B, QLoRA | 21.3 GB | fits | fits | fits |
| Llama 3.1 70B, QLoRA | 44.3 GB | no | fits | fits |
The VRAM column is identical on every card, which is the whole point: there is no such thing as an “H100-shaped” memory saving. The H100 buys you an 80GB ceiling over the A100 40GB, and nothing else in this table.
Speed is where they part. At the same 8B QLoRA configuration:
| Card | Dense bf16 TFLOPs | Estimated tokens/s | Relative |
|---|---|---|---|
| A100 40GB | 312 | 1,943 | 1.00× |
| A100 80GB | 312 | 1,943 | 1.00× |
| H100 80GB | 989 | 6,158 | 3.17× |
| H200 141GB | 989 | 6,158 | 3.17× |
That 3.17× is TFLOPs divided by the 6·N flops per token a training step costs, times a 0.3 efficiency factor for QLoRA’s dequantisation overhead. It is an estimate with two honest caveats: the model uses dense bf16 throughput, so it does not credit the H100 for FP8 or faster attention kernels, and it does not charge it for anything either. Read 3.17× as an order of magnitude, not a benchmark.
Note the A100 40GB and 80GB are the same speed here. They are the same silicon with different memory, so the 40GB is strictly the better value whenever your run fits in 40GB — which, up to 32B QLoRA, it does.
Price per GPU-hour
Marketplace median rates, which are assumptions to verify rather than quotes:
| Card | VRAM | $/GPU-hour | $ per 1M training tokens |
|---|---|---|---|
| RTX 4090 | 24 GB | 0.44 | 0.119 |
| RTX 6000 Ada | 48 GB | 0.79 | 0.195 |
| L40S | 48 GB | 1.19 | 0.147 |
| A100 40GB | 40 GB | 1.19 | 0.170 |
| A100 80GB | 80 GB | 1.59 | 0.227 |
| H100 80GB | 80 GB | 2.69 | 0.121 |
| H200 141GB | 141 GB | 3.39 | 0.153 |
The H100 costs 1.69× what an A100 80GB costs per hour, which is the number people quote when they say the H100 is expensive. The last column is the number that decides a run: the H100 lands at $0.121 per million tokens against the A100 80GB’s $0.227, because the throughput gap is bigger than the price gap. Per token, the A100 80GB is the worst value in the table. It sells memory, not compute.
Cost per fine-tune run
Take a common run: 5,000 examples, 1,500 tokens each, 3 epochs. That is 22.5M training tokens.
| Card | Hours | Cost per run |
|---|---|---|
| RTX 4090 | 6.08 | $2.68 |
| H100 80GB | 1.01 | $2.73 |
| A100 40GB | 3.22 | $3.83 |
| L40S | 2.77 | $3.30 |
| A100 80GB | 3.22 | $5.12 |
| H200 141GB | 1.01 | $3.44 |
Two things fall out of that table. First, the H100 is 47% cheaper per run than the A100 80GB despite costing 69% more per hour. Second, an 8B QLoRA run needs 6.2GB, so no card in this list is memory-constrained, and a $0.44/hour consumer card finishes within two percent of the H100 on cost. The H100 wins the clock by five hours and loses nothing on price.
Scale the model up and the pattern holds but the memory column starts to bite. For 70B QLoRA (2,000 examples × 2,000 tokens × 2 epochs, 8M tokens):
| Card | Hours | Cost per run |
|---|---|---|
| H100 80GB | 3.17 | $8.53 |
| L40S | 8.67 | $10.31 |
| H200 141GB | 3.17 | $10.76 |
| A100 80GB | 10.06 | $15.99 |
A 70B bf16 LoRA run needs 146.6GB and fits no single card in this list, so it is out of scope here: sharding changes the arithmetic and this model does not price multi-GPU.
When H100 wins, when it does not
The H100 wins when memory forces you above 24GB and you will run more than a couple of experiments. At 70B it is 47% cheaper per run, and it does in 3.2 hours what an A100 80GB does in 10. If you are iterating on a dataset, three experiments per A100 run is not a convenience, it is the difference between a day and three.
It wins on wall clock even when it loses on price. A 1-hour run versus a 6-hour run on a consumer card is the same money, but only one of them lets you fix a broken tokenizer before dinner. Time-to-result is a real cost when a person is waiting.
It wins on long sequences. Activations grow with sequence length, so per-token compute grows with it, and the faster card compounds that advantage on 8k-context data more than the estimate shows.
The H100 loses when your run fits in 24GB. Everything from 1B to 32B QLoRA does. A 4090 at $2.68 per run is not worse than an H100 at $2.73; it is the same run for the same money with five more hours to spare.
It loses on one-off jobs and on spot markets. A 10-hour checkpoints-and-resume workflow on spot A100s has more chances to be preempted than a 3-hour one, and every preemption costs the steps since your last save.
It loses when your stack is plain bf16. Part of what you pay for is FP8, faster attention and the memory bandwidth that goes with them. If your training loop cannot use those features, you still get roughly the 3× TFLOPs, but you are leaving part of the price on the table.
The short version: for 8B work the card barely matters to the bill, so pick by availability and wall clock. For 70B, the H100 or an L40S beats the A100 on cost per run, and the A100 40GB is the honest budget pick only as far as 32B.
Run your own configuration through the cost calculator before booking anything. If you are still choosing a card rather than a cloud, the GPU buying guide works from memory requirements instead of hourly rates, and QLoRA explains the 4-bit accounting that makes any of these models fit at all.