H100 vs H200: same Hopper compute, very different memory

The H100 vs H200 question is simpler than most GPU comparisons. Both are NVIDIA Hopper GPUs, and NVIDIA lists identical compute for them: 989 TFLOPS of dense BF16 and 1,979 TFLOPS of dense FP8. What changes is memory. The H200 has 141 GB of HBM3e at 4.8 TB/s, the H100 has 80 GB of HBM3 at 3.35 TB/s. The right choice depends on how much memory your model, its KV cache and your batch need, and on whether your workload waits on compute or on memory.

H100 vs H200: the short answer

Choose the H100 when everything fits in 80 GB per GPU with headroom: LoRA fine-tuning, serving models up to about 30B parameters, and training jobs already sharded comfortably across GPUs. You get the same compute for $2,190 a month instead of $3,090.

Choose the H200 when memory is the constraint: 70B-class models serving real traffic, long prompts, retrieval-augmented generation with large contexts, or many concurrent users. There, the extra 61 GB decides whether the job runs on one GPU or needs several.

What fits on one GPU

Weights take roughly 2 bytes per parameter in BF16, 1 byte in FP8 and half a byte in 4-bit formats. On top of that come the KV cache, which grows with context length and concurrency, and 10–30% headroom for activations and runtime overhead. The free memory below is simple subtraction, before KV cache and overhead:

On 8-GPU servers the totals are 640 GB against 1,128 GB. A 405B model in FP8, about 405 GB of weights, fits on either, leaving about 235 GB for KV cache and overhead on the H100 server against about 723 GB on the H200 server. In BF16, about 810 GB, it fits only on the H200 server. The VRAM calculator runs the same math for your exact model, precision and context length.

Long contexts and the KV cache

Weights are only part of the memory bill. During inference, every token of every active conversation keeps its keys and values in GPU memory. Take a common 70B layout with grouped-query attention (80 layers, 8 KV heads of dimension 128): the KV cache needs about 0.33 MB per token in BF16, or about 0.16 MB in FP8.

• One H100 with the model in FP8 keeps at most about 5 GB for the KV cache once overhead is counted: roughly 30,000 tokens in FP8, shared by every concurrent request.
• One H200 keeps roughly 55–60 GB: more than 300,000 tokens in FP8.

A single 128K-token context needs about 21 GB of FP8 KV cache, which fits beside FP8 weights on an H200 but not on an H100. When the cache is full, inference engines queue or preempt requests, so concurrency drops even though the GPU still has compute to spare. For 70B-class serving, this is usually the deciding factor.

Memory bandwidth and token generation

Generating each new token means reading the model's weights, plus the KV cache of each request, from memory. At small and medium batch sizes, decoding is therefore memory-bound: the tensor cores wait on HBM. The H200 moves 4.8 TB/s against 3.35 TB/s on the H100, 43% more bytes per second with the same compute, so it gains most exactly where the H100 is waiting on memory.

Prefill, the processing of the prompt, is mostly compute-bound, and there the two GPUs are close. Workloads dominated by long prompts and short answers, such as classification or extraction, gain less from the H200 than chat, agents or code generation with long outputs.

Training, fine-tuning and 8-GPU servers

For training the gap narrows further: same tensor cores, same NVLink 4 at 900 GB/s per GPU, and in our catalog the same host. Both HGX servers come with two Xeon 8480+ CPUs (112 cores), 2 TB of DDR5, 8× 3.84 TB NVMe and 8× 400G ConnectX-7. If a training job already fits on 8× H100 with a sensible batch size, the H100 server does the same work for $7,000 less per month.

Fine-tuning follows the same logic. QLoRA on a 70B model loads the weights in 4 bits, about 35 GB, so it runs on a single H100 with room for activations. LoRA on the same model in BF16 keeps about 140 GB of frozen weights in memory, which does not fit on one GPU of either kind; plan an 8-GPU server for that job.

Memory still matters at scale. Full fine-tuning with Adam needs roughly 16–20 bytes per parameter before activations, so a 30B model needs about 480–600 GB: that fits on 8× H100 (640 GB) with little left for activations, and comfortably on 8× H200 (1,128 GB). For serving, an 8× H200 server holds a 671B-parameter mixture-of-experts model in FP8, about 671 GB of weights, on one machine; 640 GB of H100 memory cannot. For that class of model, see our DeepSeek hardware requirements guide; both servers are listed under 8-GPU servers.

Cost per month and value

One H200 costs $3,090 a month against $2,190 for an H100: $900 more, or 41%. On 8-GPU servers it is $23,900 against $16,900, also 41% more. What the premium buys:

• Memory 76% more capacity, so the H200 costs less per GB: about $21.91 against $27.38 a month.
• Bandwidth 43% more, roughly in line with the price: about $644 against $654 a month per TB/s.
• Compute nothing extra, so for compute-bound jobs the H100 gives more vendor-listed FLOPS per dollar.

A worked example: 30 concurrent conversations of 8K tokens with a 70B model in FP8 need about 40 GB of KV cache. One H100 holds three or four of them at a time; one H200 holds all 30 with room to spare. Splitting the model across two single-GPU H100 servers is not practical, since separate servers do not share NVLink, so the next H100 step is a full 8-GPU server at $16,900. For this workload, the H200 at $3,090 is clearly the cheaper option.

How to decide

1. Estimate weights: parameters × bytes per parameter at the precision you will run.
2. Add the KV cache for your longest context times your target concurrency.
3. Add 10–30% headroom. Under about 70 GB, start with the H100; between 70 and 125 GB, take the H200; above that, see our H200 vs B200 comparison.

Both GPUs run the same CUDA builds, so moving between them needs no code changes, and our Ubuntu images ship with NVIDIA drivers, CUDA, Docker and the NVIDIA Container Toolkit preinstalled. Both are in stock as single GPUs and as 8-GPU servers, and every order gets a delivery date confirmed in writing before payment. For sizing a production deployment, see inference on dedicated GPUs.