GPU memory (VRAM) calculator for LLMs

Our LLM VRAM calculator answers the first question of any GPU decision: how much VRAM do you need to run a model, or to fine-tune it? The estimate adds up what occupies GPU memory (the model weights, the KV cache and the runtime overhead) and, for training, the gradients and optimizer states on top. This page explains the formulas, so you can check a result or redo the math for any model.

How the LLM VRAM calculator works

For inference, the estimate is:

GPU memory ≈ weights + KV cache + overhead (10–20%)

The weights are fixed once you pick a model and a precision. The KV cache grows with context length and with the number of requests served at the same time. The overhead covers everything else the runtime allocates. At short contexts the weights dominate; at long contexts or high concurrency, the KV cache can take over. Fine-tuning adds gradients and optimizer states, which is why training a model needs several times the memory of serving it.

Model weights: parameters × bytes per parameter

Each parameter is stored in a numeric format, and the format sets its size:

Quantized formats also store scaling factors, and some layers are often kept at higher precision, so real 4-bit checkpoints come out a few percent larger. Lower precision saves memory and bandwidth at an accuracy cost that depends on the model and the method; our guide to FP8 and FP4 precision covers the trade-off.

KV cache: memory that grows with every token

During generation, the model keeps the attention keys and values of every token in context, so it does not recompute them. Per token, that costs:

KV cache per token = 2 × layers × KV heads × head dimension × bytes per value

The factor 2 counts keys and values. Multiply by the tokens in flight: context length × concurrent sequences.

Worked example: a 70B model with grouped-query attention

Take a 70B model with 80 layers, 8 KV heads and a head dimension of 128, with the cache in BF16 (2 bytes per value):

• 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or about 320 KB per token.
• One sequence of 32k tokens (32,768): about 10 GB.
• Eight concurrent 32k sequences: about 80 GB, more than the model's FP8 weights.

Grouped-query attention is what keeps this manageable: the 8 KV heads are shared by all the query heads. A layout with one KV head per query head (64 in a typical 70B model) would need eight times as much. Storing the cache in FP8 halves it, and capping context length or concurrency in your inference server bounds it.

Runtime overhead and activations

Besides weights and cache, the GPU holds the CUDA context, the inference engine's workspace, activations for the batch in progress, CUDA graphs if enabled, and some fragmentation. We add 10–20% for all of this: the low end for a lean server with short prompts, the high end for large batches, long prefills or several processes on one GPU.

Expect monitoring tools to show a nearly full GPU even at idle. Inference servers such as vLLM reserve a fixed share of memory at startup and fill it with KV cache blocks; the size of that reserve decides how many requests run at once.

Fine-tuning memory: full, LoRA and QLoRA

• Full fine-tuning with AdamW in mixed precision needs roughly 16 bytes per parameter before activations: 2 for BF16 weights, 2 for gradients, 4 for an FP32 master copy and 8 for the two optimizer moments. For a 70B model that is about 1.1 TB, a job for an 8-GPU server with sharding (FSDP or ZeRO-3), such as 8× B200 (1.44 TB) or 8× B300 (2.3 TB).
• LoRA freezes the base model and trains small adapter matrices, usually well under 1% of the parameters. Memory is roughly the frozen weights (about 140 GB for 70B in BF16) plus the adapters, their gradients and optimizer states, plus activations.
• QLoRA loads the frozen weights in 4-bit, about 0.5 bytes per parameter, so a 70B base model takes roughly 35–40 GB before adapters and activations. One 80 GB H100 or 96 GB RTX PRO 6000 handles it at moderate sequence lengths.

Activations grow with batch size and sequence length and can exceed the weights on long sequences. Gradient checkpointing trades extra compute for much less activation memory.

Splitting a model across several GPUs

When the total exceeds one GPU, tensor parallelism splits each layer's weights and the KV cache across GPUs, so each holds about 1/N of both, plus its own overhead. Keep about 10% headroom per GPU. A 70B model in BF16 on an 8× H100 server (640 GB in total) puts about 17.5 GB of weights on each 80 GB GPU, which leaves most of the memory for KV cache and long contexts. Tensor parallelism exchanges data at every layer, so it runs best on NVLink-connected GPUs, as in our NVIDIA HGX 8-GPU servers.

Limits of any VRAM estimate

• Mixture-of-Experts models run only a few experts per token, but every expert must sit in memory. Size them by total parameters, not active parameters.
• Framework overheads vary with the engine, its version, kernels and settings. Two servers running the same model can differ by several gigabytes.
• Long contexts shift the balance: from 128k tokens up, the KV cache can outgrow the weights. Attention variants such as sliding-window or latent attention change the per-token formula.
• Multimodal models add vision or audio encoders and their activations.

Treat the result as a planning figure and confirm it with a real load test. For a walk-through by model size, read our guide to VRAM for LLMs.

Which GPU fits your estimate

With 10% headroom, these are the largest estimates each single GPU can hold:

Above that, an 8-GPU server holds up to 2.3 TB (8 × 288 GB on HGX B300 or AMD Instinct MI355X). Each GPU page, such as the H100, H200, B200 and B300, lists what fits in its memory, or you can browse every GPU we rent.