How much VRAM do I need for an LLM?

How much VRAM do I need for an LLM? Roughly: the model's weights, plus a KV cache that grows with context length and with the number of requests served at once, plus 10–30% for activations and runtime overhead. Training adds gradients and optimizer states on top. This guide gives you the formulas, works through real numbers for Llama 3.1 8B and 70B, and maps the results to GPUs from 32 GB to 288 GB. If you would rather skip the arithmetic, the VRAM calculator does it for you.

How much VRAM you need for an LLM: the formula

For inference, memory breaks down into three parts:

VRAM     ≈ weights + KV cache + overhead
weights  = parameters × bytes per parameter
KV cache = 2 × layers × KV heads × head dim × bytes per value × tokens in flight
overhead = activations, CUDA context, engine buffers (plan 10-30%)

Bytes per parameter are 2 for BF16 or FP16, 1 for FP8 or INT8, and 0.5 for 4-bit formats. "Tokens in flight" means context length times the number of sequences served concurrently. All sizes in this article are decimal gigabytes (one billion bytes) and deliberately rough: real checkpoints keep some layers at higher precision, and inference engines allocate memory in blocks.

Model weights by precision

Weights are the floor. If they do not fit, nothing else matters. Multiply parameters by bytes per parameter:

Treat the FP8 and 4-bit columns as minimums. Quantized formats store scaling factors next to the weights, and quantized checkpoints usually keep the embedding and output layers in 16-bit. Those layers weigh more in small models: in Llama 3.1 8B, the two matrices hold about 1.05 billion parameters, roughly 2.1 GB in BF16, which is more than half of what the rest of the model takes at 4-bit. Precision also affects quality, more at 4-bit than at FP8; our explainer on FP8 and FP4 precision covers the trade-offs.

KV cache: the memory that grows with context

During generation, the model keeps a key vector and a value vector for every token, in every layer, for every sequence it serves. That is the KV cache. With long contexts or many concurrent users, it can outgrow the weights.

KV bytes per token = 2 × layers × KV heads × head dim × bytes per value
KV cache           = KV bytes per token × context length × concurrent sequences

The KV-heads term is where modern architectures save memory. Llama 3.1 uses grouped-query attention: the 70B model has 64 query heads but only 8 key-value heads, so its cache is one eighth of what full multi-head attention would need. With the published architectures and a 16-bit cache:

• Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128): 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, about 0.13 MB per token.
• Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128): 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, about 0.33 MB per token.

Three lessons follow. One 128K-token conversation on the 70B model needs about 43 GB of cache, more than half of an H100. Sixteen users at 8K tokens need exactly as much as that single long conversation, because what counts is the total number of tokens in flight. And an FP8 cache halves every number in the table, which is why inference engines such as vLLM offer it as an option.

The table is a worst case. Engines with a paged KV cache allocate blocks as tokens are generated, so a request that stops early does not hold its full context. Size for the realistic peak of tokens in flight rather than for every user hitting the maximum at once, but make sure at least one maximum-length request fits.

Overhead and headroom

Weights and cache are not the whole bill. Before your model loads, the CUDA context and libraries take memory on every GPU, commonly a few hundred megabytes up to a gigabyte or two. During a forward pass, activations need working space, which peaks when long prompts are processed. Inference engines also reserve memory for CUDA graphs and sampling buffers, and multi-GPU jobs add communication buffers.

The practical rule is to plan 10–30% headroom over weights plus KV cache: the low end for one GPU running a well-understood engine, the high end for multi-GPU setups, very long prompts or training. vLLM makes this explicit. It takes a fixed fraction of each GPU's memory (90% by default, set with --gpu-memory-utilization), loads the weights, measures peak activation memory with a test pass, and gives everything that remains to the KV cache. Our guide to self-hosting LLM inference with vLLM shows how to tune it.

Worked example: serving Llama 3.1 70B

Say you want to serve Llama 3.1 70B to 16 concurrent users with up to 8,192 tokens each.

• Weights in FP8: about 70 GB, a little more in practice.
• KV cache in BF16: 16 × 8,192 tokens × 0.33 MB ≈ 43 GB.
• Total with 15% headroom: (70 + 43) × 1.15 ≈ 130 GB.

That fits a single 141 GB H200. It does not fit a single 80 GB H100: the weights take almost all of it, leaving a few gigabytes for the cache, enough for one or two 8K requests at a time rather than sixteen. Halving the cache with FP8 brings the total to about 105 GB, which still needs more than one H100.

In BF16, the weights alone are about 140 GB. That rules out a single H200 in practice, since nothing would be left for the cache. A 180 GB B200 holds the weights with roughly 20 GB to spare after overhead, enough for about six full 8K requests, or about twelve with an FP8 cache. A 288 GB B300 serves all sixteen with room to spare.

Fine-tuning memory: full, LoRA and QLoRA

Training needs much more memory per parameter than inference, because the GPU also holds gradients and optimizer states. The figures below are before activations, which depend on sequence length and micro-batch size and are usually kept in check with activation (gradient) checkpointing.

In practice, LoRA on an 8B model fits a 32 GB RTX 5090 or a 48 GB L40S with room for activations. Full fine-tuning of the same model needs a 180 GB B200 or a 288 GB B300, or several 80 GB GPUs with sharded optimizer states. At 70B, QLoRA fits a single H100 or H200, LoRA needs two 80 GB GPUs or one B200 or B300, and full fine-tuning needs an 8-GPU B300 server, an 8× B200 with a lean setup, or 8× H200 with optimizer states offloaded to CPU memory. Our guide on how to fine-tune a 70B model works through each case.

MoE models: every expert must be in memory

Mixture-of-experts models route each token through a few experts out of many. That cuts the compute per token, not the memory: every expert must stay resident, because the next token may need any of them. Size MoE models by total parameters, and treat active parameters as a guide to compute only.

• Mixtral 8x7B has 46.7B parameters in total but uses about 12.9B per token. In BF16 it still needs about 93 GB for its weights, more than one H100.
• Qwen3-235B-A22B has 235B parameters with 22B active: about 470 GB in BF16, which fits on 8× H100 (640 GB), or about 235 GB in FP8.
• DeepSeek-V3 has 671B parameters with 37B active. Even in FP8 its weights take roughly 670 GB: more than the 640 GB of an 8× H100 server, but well within 8× H200 (1,128 GB) or 8× B200 (1,440 GB).

Attention design still drives the cache. DeepSeek's models use multi-head latent attention, which stores a compressed cache, so the formula above overestimates it for them. For that family, see our guide to DeepSeek hardware requirements.

Which GPU fits your model

Here is how the numbers map to real hardware for inference, assuming moderate context and concurrency. "Tight" means the weights fit but leave little room for cache.

Two notes on multi-GPU setups. Splitting a model with tensor parallelism splits its KV cache too, so cache capacity grows with the GPU count. And the interconnect matters: on HGX servers the GPUs share NVLink, while RTX PRO 6000 and L40S cards talk over PCIe, which is fine for running separate copies of a model and slower for splitting one model across cards. For a ranked view by workload, see our guide to the best GPU for LLM inference.

Everything above assumes you get the whole card. Every GPU we rent is a whole physical GPU, so the full memory in this table is yours, with no vGPU profile or MIG partition to subtract. If 70B in FP8 with real concurrency is your target, the NVIDIA H200 page shows the single-GPU and 8-GPU options.

Frequently asked questions

How much VRAM do I need to run a 70B model?

About 140 GB for the weights in BF16, 70 GB in FP8 and 35–40 GB in 4-bit, plus the KV cache and 10–30% overhead. In practice that means a B200 or B300 for BF16, an H200 for FP8 with real concurrency, and an 80 GB GPU for 4-bit.

How much VRAM does an 8B model need?

About 16 GB for the weights in BF16, 8 GB in FP8 and 4–6 GB in 4-bit. Add the KV cache: for Llama 3.1 8B that is about 0.13 MB per token in BF16, or roughly 1 GB per 8,000 tokens.

Does context length really change VRAM usage?

Yes. The KV cache grows linearly with context length and with the number of concurrent sequences. One 128K-token request on Llama 3.1 70B needs about 43 GB of cache in BF16, more than half of an H100.

Can I combine the memory of several GPUs?

Yes. Inference engines and training frameworks can split a model across GPUs with tensor or pipeline parallelism, and the KV cache is split too. This works best over NVLink, as on 8-GPU HGX servers; over PCIe it works, but communication costs more.

Is all of the advertised GPU memory usable?

Not quite: the CUDA context, libraries and framework buffers take some memory on every GPU, and inference servers reserve working space for activations. On a shared or partitioned GPU you may only get a slice, so check that you are renting the whole card.