Best GPU for LLM inference in 2026: a sizing guide by model size

The best GPU for LLM inference is the least expensive one that holds your model's weights at the precision you serve, leaves room for the KV cache your traffic needs, and has enough memory bandwidth to hit your latency target. Compute matters too, but later than most spec sheets suggest. This guide walks through those checks in order, then maps common model sizes to the GPUs in our catalog, from a single RTX 5090 to 8-GPU servers.

How to choose the best GPU for LLM inference

Make the decisions in this order. Each step narrows the list for the next one.

1. Precision. BF16, FP8 or 4-bit sets the bytes per parameter.
2. Weights. They must fit, with headroom.
3. KV cache. Context length × concurrent sequences, often the deciding factor between two candidate GPUs.
4. Memory bandwidth. It sets decode speed.
5. Compute. It matters for long prompts and large batches.
6. Interconnect. Only if the model needs more than one GPU.
7. Price. Per month, per GB and per unit of bandwidth.

Step 1: model size and precision set the memory floor

Weights take roughly parameters × bytes per parameter: 2 bytes in BF16 or FP16, 1 byte in FP8 or INT8, and about half a byte in 4-bit formats (a little more once block scales are counted). A 70B model therefore needs about 140 GB in BF16, 70 GB in FP8 and 35–40 GB in 4-bit. Plan 10–30% on top for the KV cache, activations and runtime.

Precision is a quality decision as much as a memory one. FP8 is widely used for serving because it halves memory and its quality cost is often small; 4-bit saves more but needs careful validation on your own tasks. Our explainer on FP8 vs FP4 vs BF16 covers the trade-offs, and the VRAM calculator runs the numbers for a specific model.

Step 2: KV cache and concurrency

For every token in every active sequence, the model keeps a key vector and a value vector in each layer. The size per token is:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

Take a 70B-class model with grouped-query attention: 80 layers, 8 KV heads of 128 dimensions, cache in BF16. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 0.33 MB per token.

Now put it next to the weights. On one H200 with FP8 weights (about 70 GB), roughly 50–60 GB remains for the cache after runtime overhead: about 20 concurrent 8K-token sequences with a BF16 cache, or about 40 with an FP8 cache. On one H100, the same weights leave only a few GB, enough for a handful of short requests. Both GPUs "run the model"; only one serves real traffic comfortably. Engines such as vLLM allocate the cache in pages as tokens arrive, so average lengths matter more than the maximum, and they report their KV capacity at startup.

Step 3: why memory bandwidth drives decode speed

Inference has two phases. Prefill processes the whole prompt at once: large matrix multiplications, limited mainly by compute. Decode then generates one token per sequence per step, and every step must stream all active weights from memory, plus each sequence's KV cache, to perform roughly two floating-point operations per weight per sequence. At small and moderate batch sizes, the arithmetic units finish long before the next bytes arrive. Decode speed per sequence is therefore capped by bytes per second, not by FLOPS.

Four consequences follow:

• Bandwidth ratios bound decode gains. At the same batch size, decode on an H200 (4.8 TB/s) can be at most about 1.4× faster than on an H100 (3.35 TB/s), and on a B200, B300 or MI355X (8 TB/s) about 2.4×. Larger memory adds to that by allowing larger batches.
• Lower precision helps twice. FP8 weights halve the bytes read per step as well as the memory used, so decode speeds up in the memory-bound regime.
• Batching amortizes weight reads. One pass over the weights serves every sequence in the batch, so throughput rises with batch size until compute or KV cache capacity runs out. That is why spare memory for the cache turns directly into throughput.
• MoE models read less per token. Only the selected experts are read for each token, so decode costs less than for a dense model of the same total size, but all experts must still fit in memory.

Model size to GPU: our recommendations

This table maps common model sizes to GPUs in our catalog. "Recommended" leaves room for a useful KV cache at moderate context lengths; "Also works" lists options that are tighter, slower or larger than needed. Sizes are for dense models unless noted.

Two patterns stand out. The RTX PRO 6000 covers a wide range: 96 GB holds anything up to about 32B in BF16 or 70B in FP8. And in our catalog the step from one GPU to eight is a big one: there is nothing between a single B300 (288 GB) and an 8-GPU server, so models with roughly 250 GB to 600 GB of weights go straight to a full server. Our guide to DeepSeek R1 hardware requirements works through the largest case in detail.

Cost per month per GB of GPU memory

Once you know how much memory you need, compare the price per GB. Bandwidth belongs in the same table, because the cheapest memory is usually the slowest.

How to read it:

• Price per GB matters most when the model barely fits. If you are paying for capacity to hold weights and cache, the 8× MI355X server ($8.64 per GB, if your stack runs on ROCm) and the RTX PRO 6000 configurations (about $14) stand out.
• Bandwidth matters when the model fits easily. A 32B model fits on both an RTX PRO 6000 and an H100; the H100 costs more per GB but streams weights about twice as fast.
• Per GB, a single B300 costs less than a B200. It has 60% more memory for about 13% more per month, in the same 8 TB/s bandwidth class. Its stock is limited.
• Eight GPUs in one server cost a little less than eight single-GPU plans (8× H100 is $16,900 against 8 × $2,190 = $17,520), but you take all eight at once.

When to use 8 GPUs with tensor parallelism

Tensor parallelism splits every layer's weight matrices across GPUs. Each GPU reads only its slice of the weights at each decode step, so eight GPUs bring eight times the memory bandwidth to every token, as well as eight times the memory. The cost is communication: two all-reduce operations per transformer layer at every step, which is why tensor parallelism belongs on GPUs connected by NVLink. Our HGX servers provide 900 GB/s per GPU on H100 and H200 (NVLink 4) and 1.8 TB/s on B200 and B300 (NVLink 5). The 8× RTX PRO 6000 server connects its GPUs over PCIe Gen5 without NVLink, so prefer independent replicas or pipeline parallelism there. Our article on NVLink vs PCIe works through the traffic.

Use tensor parallelism when the weights plus the KV cache you need do not fit on one GPU, or when you need lower per-request latency on a large model than one GPU can deliver. Otherwise, use the smallest tensor-parallel degree that fits and add replicas for throughput. A 70B model in FP8 on an 8× H200 server can run as eight single-GPU replicas, as four replicas of two GPUs each with more cache per replica, or as one eight-GPU replica, which usually gives the lowest latency per request. Measure all three with your own traffic.

# Two replicas of four GPUs each on one 8-GPU server (vLLM)
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve <model> --tensor-parallel-size 4 --port 8000
CUDA_VISIBLE_DEVICES=4,5,6,7 vllm serve <model> --tensor-parallel-size 4 --port 8001

Quick picks

• Prototyping, small models and embeddings: RTX 5090 or L40S.
• Production up to about 32B, or 70B in FP8 on a budget: RTX PRO 6000.
• 70B in FP8 with room for long contexts: H200.
• 70B in BF16 or 120B-class in FP8: B200 or B300. Among single GPUs, only the B300 holds a 4-bit 405B model.
• 405B dense or 671B MoE: 8× H200, 8× B200, 8× B300 or 8× MI355X.
• Many replicas of mid-size models: 8× RTX PRO 6000.

Every configuration above is a whole, dedicated GPU at a flat monthly price. If your case sits between two rows, ask our engineers to size it with you; for the serving side, see our guide to self-hosting LLM inference with vLLM.

Frequently asked questions

What is the best GPU for LLM inference in 2026?

It is the least expensive GPU that fits your weights plus KV cache with enough bandwidth for your latency target. As a rule of thumb, an RTX PRO 6000 covers models up to about 32B, an H200 serves 70B in FP8 comfortably, a B200 or B300 handles 70B in BF16, and 8-GPU servers cover 405B and 671B-class models.

How much GPU memory do I need to run a 70B model?

About 140 GB for the weights in BF16 or about 70 GB in FP8, plus the KV cache and 10 to 30 percent headroom. In FP8 that means one H100 tightly or one H200 comfortably.

Is memory bandwidth more important than compute for inference?

For decode at small and moderate batch sizes, yes: every step streams the active weights from memory, so bandwidth caps per-sequence speed. Compute matters more for the prefill of long prompts and for very large batches.

Do I need NVLink to serve a model across several GPUs?

Not strictly, but tensor parallelism exchanges data at every layer, so it runs best over NVLink. Without NVLink, prefer pipeline parallelism, or independent replicas of a model that fits on one GPU.

Should I serve in FP8 or BF16?

FP8 halves weight memory and the bytes read per token compared with BF16, and its quality cost is often small, but validate it on your own evaluation set. Hopper, Ada Lovelace, Blackwell and the MI355X all support FP8 in hardware.