LLM GPU requirements
GPU requirements for open LLMs
How much VRAM popular open models need to serve and fine-tune, and the cheapest servers that run them. Every number comes from the model’s published architecture and the same formulas as our VRAM calculator.
- Configurations
- 15
- Available now
- 13
- From
- $449/mo
- Setup & egress fees
- $0
| Model | Parameters | Weights FP8 | KV cache per 1k tokens | To serve 8k × 8 requests | Cheapest server |
|---|---|---|---|---|---|
| Llama 3.1 8BMeta · Dense | 8B | 8.0 GB | 131 MBBF16 | 27 GBBF16 | $449NVIDIA GeForce RTX 5090 |
| Qwen3 14BAlibaba Qwen · Dense | 14.8B | 15 GB | 164 MBBF16 | 44 GBBF16 | $1,390NVIDIA RTX PRO 6000 Blackwell |
| Mistral Small 24BMistral AI · Dense | 23.6B | 24 GB | 164 MBBF16 | 38 GBFP8 | $890NVIDIA L40S |
| Qwen3 32BAlibaba Qwen · Dense | 32.8B | 33 GB | 262 MBBF16 | 54 GBFP8 | $1,390NVIDIA RTX PRO 6000 Blackwell |
| Llama 3.3 70BMeta · Dense | 70.6B | 71 GB | 328 MBBF16 | 100 GBFP8 | $3,090NVIDIA H200 |
| gpt-oss-120bOpenAI · Mixture of Experts | 117B5.1B active | 117 GB | 37 MBBF16 | 68 GBMXFP4 (native 4-bit) | $1,390NVIDIA RTX PRO 6000 Blackwell |
| Qwen3 235B-A22BAlibaba Qwen · Mixture of Experts | 235B22B active | 235 GB | 193 MBBF16 | 272 GBFP8 | $10,4908× NVIDIA RTX PRO 6000 Blackwell · PCIe, no NVLink |
| Llama 3.1 405BMeta · Dense | 405B | 405 GB | 516 MBBF16 | 480 GBFP8 | $10,4908× NVIDIA RTX PRO 6000 Blackwell · PCIe, no NVLink |
| DeepSeek-V3 / R1DeepSeek · Mixture of Experts | 671B37B active | 671 GB | 70 MBMLA, BF16 | Read the DeepSeek hardware guide | |
Swipe the table to see every column
Serving totals include the weights, a BF16 KV cache for 8 requests of 8k tokens and about 10% runtime overhead, with 8% of GPU memory kept free. Models split across GPUs run best on NVLink servers; the 8× RTX PRO 6000 server links its GPUs over PCIe. Prices in USD per month, excluding VAT.
Model guides
- Meta Llama 3.1 8B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Alibaba Qwen Qwen3 14B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Mistral AI Mistral Small 24B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Alibaba Qwen Qwen3 32B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Meta Llama 3.3 70B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- OpenAI gpt-oss-120b GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Alibaba Qwen Qwen3 235B-A22B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
- Meta Llama 3.1 405B GPU requirements VRAM in BF16, FP8 and 4-bit, KV cache by context, fine-tuning memory and the servers that run it. See the sizing
Talk to an engineer
Not sure which server fits your model?
Tell us what you need. A GPU engineer replies with availability, a confirmed delivery date and a payment link.
- Delivery date confirmed before you pay
- Cancel any month
- Card or crypto, same price