GPU Memory Planning for LLMs

the four line items

Component	Scales with	Covered on
Model weights	Parameter count × bytes per parameter	Quantization
KV cache	Sequence length × concurrent requests × layers/heads	KV Cache & Continuous Batching
Activations	Batch size × sequence length, transient during a forward pass	this page
Optimizer state (training only)	Trainable parameter count × ~2-8× (Adam-family)	LoRA & PEFT

a worked inference example

# serving a 13B-parameter model at INT4, batch of 8, 4k context each
weights_gb   = 13 * 0.5                       # INT4 ≈ 0.5 bytes/param -> 6.5 GB
kv_per_tok   = 2 * 40 * 8 * 128 * 2           # layers=40, kv_heads=8 (GQA), head_dim=128, FP16
                                              # -> 163,840 bytes per token
kv_gb        = (kv_per_tok * 4096 * 8) / 1e9  # 4096 tokens x 8 concurrent requests
# kv_gb ≈ 5.4 GB

activation_overhead_gb = 2   # rough headroom, workload-dependent

total_gb = weights_gb + kv_gb + activation_overhead_gb
print(total_gb)   # ≈ 13.9 GB -- fits comfortably on a single 24GB card

# now raise the serving target: 32 concurrent requests at 8k context each
kv_gb_busy = (kv_per_tok * 8192 * 32) / 1e9
# kv_gb_busy ≈ 42.9 GB -- the same weights, but KV cache is now 6.6x them
# total ≈ 51.4 GB: no longer a single-24GB-card problem

At this modest serving target the weights are still the bigger line item — but notice how differently the two terms scale. Weight memory is fixed once you pick the model and the quantization. KV cache grows linearly with both context length and concurrency, so the second calculation above multiplies it by 8 while the weights never move. This is the part that surprises people who size a GPU from parameter count alone: the number that decides whether you fit is usually the one that depends on your traffic, not your model.

fine-tuning needs a very different budget

Training (even LoRA) adds gradient and optimizer-state memory that inference never touches. For full fine-tuning, Adam's optimizer state alone typically adds 2 extra FP32 copies of every trainable parameter (momentum and variance) on top of the weights and gradients — roughly 4x the raw parameter memory before activations are even counted, which is precisely the cost LoRA exists to avoid by keeping the trainable parameter count tiny.

levers, if the number doesn't fit

Lever	Reduces
Quantize weights (INT8/INT4)	Weight memory directly
Lower max context length or concurrent request cap	KV cache, often the largest lever at high concurrency
Grouped-query attention (a model architecture choice, not a runtime one)	KV cache per token, by using fewer KV heads than query heads
LoRA instead of full fine-tuning	Optimizer state and gradient memory, training only
Multi-GPU tensor/pipeline parallelism	Per-GPU share of weights (spreads the model, doesn't reduce the total)

where to go from here

Quantization — line item 1 in the budget above.
KV Cache & Continuous Batching — line item 2, and why it often dominates.
Anatomy of a GPU — the hardware this budget has to fit inside.