LLM GPU requirements
Llama 3.3 70B GPU requirements
How much GPU memory Llama 3.3 70B needs in BF16, FP8 and 4-bit, how its KV cache grows with context, what fine-tuning takes, and which servers run it, with monthly prices.
- Parameters
- 70.6B
- Architecture
- Dense 80 layers
- KV heads
- 8 × 128 dim
- Context
- 128k tokens
Cheapest servers that run it
FP8 weights · 8k context · 8 concurrent requests · about 100 GB
-
NVIDIA H200 In stock
141 GB HBM3e · Single GPU
$3,090/mo
-
NVIDIA B200 In stock
180 GB HBM3e · Single GPU
$4,590/mo
-
NVIDIA B300 Limited
288 GB HBM3e · Single GPU
$5,190/mo
Delivery date confirmed before you pay. Missed it? Full refund.
Inference memory
How much VRAM Llama 3.3 70B needs
Weights take parameters × bytes per parameter. Serving adds the KV cache and about 10% runtime overhead; the totals below assume an 8k context with 8 concurrent requests and a BF16 KV cache.
| Precision | Weights | Total to serve | Cheapest server that fits |
|---|---|---|---|
| BF16 2 bytes per parameter | 141 GB | 178 GB | NVIDIA B300 $5,190/mo |
| FP8 1 byte per parameter | 71 GB | 100 GB | NVIDIA H200 $3,090/mo |
| 4-bit 0.5 bytes per parameter | 35 GB | 61 GB | NVIDIA RTX PRO 6000 Blackwell $1,390/mo |
Estimates with about 8% of GPU memory kept free. Quantized weights also keep some layers in higher precision, and frameworks differ in overhead; check the free memory your engine reports at startup. A model split across GPUs with tensor parallelism runs best on NVLink servers (HGX); the 8× RTX PRO 6000 server links its GPUs over PCIe.
KV cache
Llama 3.3 70B KV cache by context length
Each token in flight stores keys and values for every layer: 2 × 80 layers × 8 KV heads × 128 dimensions × 2 bytes = 320 KB per token in BF16. An FP8 KV cache halves these figures.
| Context per request | 1 request | 8 requests | 32 requests |
|---|---|---|---|
| 4k tokens | 1.3 GB | 11 GB | 43 GB |
| 32k tokens | 11 GB | 86 GB | 344 GB |
| 128k tokens | 43 GB | 344 GB | 1,374 GB |
Fine-tuning
Memory to fine-tune Llama 3.3 70B
Rough totals before tuning batch size and sequence length: QLoRA keeps 4-bit base weights, LoRA keeps BF16 weights frozen, and a full fine-tune with AdamW needs about 16 bytes per parameter plus activations, sharded across GPUs.
| Method | Approximate memory | Cheapest server that fits |
|---|---|---|
| QLoRA (4-bit base) | 53 GB | NVIDIA RTX PRO 6000 Blackwell $1,390/mo |
| LoRA (BF16 base) | 175 GB | NVIDIA B300 $5,190/mo |
| Full fine-tune (AdamW, mixed precision) | 1,356 GB | 8× AMD Instinct MI355X $19,900/mo |
Full fine-tuning assumes sharded training (FSDP or ZeRO-3) with activation checkpointing. See the fine-tuning guide for methods and settings.
About Llama 3.3 70B
Llama 3.3 70B is Meta’s 70B dense instruction model: 80 layers, grouped-query attention with 8 KV heads and a 128k-token context window. It is the reference point for 70B-class serving: in BF16 the weights alone take about 141 GB, more than a single H200 can hold once the KV cache is added, while FP8 halves that and fits one H200 with room to spare.
Every figure on this page comes from the model’s published architecture and the same formulas as our VRAM calculator, where you can change the context length, concurrency and KV cache precision.
Model facts
- Developer
- Meta
- License
- Llama 3.3 Community License
- Type
- Dense
- Parameters
- 70.6B
- Layers
- 80
- Attention
- 64 query heads, 8 KV headsgrouped-query attention, head dimension 128
- Context window
- 128k tokens
FAQ
Llama 3.3 70B GPU FAQ
The short version. Anything else, our engineers reply 24/7.
Ask an engineerHow much VRAM does Llama 3.3 70B need?
About 141 GB for the weights in BF16, 71 GB in FP8 and 35 GB in 4-bit, plus the KV cache: about 0.3 GB per 1,000 tokens in flight with a BF16 cache. Serving it in FP8 with an 8k context and 8 concurrent requests takes about 100 GB in total.
Can Llama 3.3 70B run on a single GPU?
Yes. In FP8, a single NVIDIA H200 (141 GB) serves it with an 8k context and 8 concurrent requests. Longer contexts or more users need more memory for the KV cache, so check your case in the VRAM calculator.
What is the cheapest FullGPU server for Llama 3.3 70B?
For FP8 serving with an 8k context and 8 concurrent requests, the NVIDIA H200 at $3,090 per month is the lowest-priced configuration that fits. For other contexts and concurrency, size it in the VRAM calculator or ask an engineer.
How much memory do I need to fine-tune Llama 3.3 70B?
Roughly 53 GB with QLoRA, 175 GB with LoRA on BF16 weights, and 1,356 GB for a full fine-tune with AdamW in mixed precision, before tuning batch size and sequence length. Full fine-tunes of large models are sharded across GPUs with FSDP or ZeRO-3.
Talk to an engineer
Run Llama 3.3 70B on your own GPUs.
Tell us what you need. A GPU engineer replies with availability, a confirmed delivery date and a payment link.
- Delivery date confirmed before you pay
- Cancel any month
- Card or crypto, same price