The right GPU for LLM fine-tuning, from QLoRA to full fine-tunes

The right GPU for LLM fine-tuning is the smallest one that holds your base model, your method's training state and your longest sequences, with headroom to spare. Below is the memory math for QLoRA, LoRA and full fine-tuning of 8B, 32B and 70B models, which single GPU handles each, and when you need an 8-GPU node instead. All of our GPUs are whole physical cards, rented by the month.

How much memory each fine-tuning method needs

• Full fine-tuning trains every weight. With AdamW in mixed precision, plan roughly 16–20 bytes per parameter before activations. 8-bit optimizer states save about 6 bytes per parameter.
• LoRA freezes the base model in BF16, at 2 bytes per parameter, and trains small adapter matrices whose optimizer state usually takes a few GB at most.
• QLoRA loads the frozen base in 4-bit, about 0.5 bytes per parameter plus some overhead, and trains BF16 adapters on top. Compare its results with LoRA on your own evaluation set before you commit.

These figures cover weights or training state only. Activations come on top and grow with sequence length and batch size; gradient checkpointing cuts them sharply in exchange for extra compute. Plan 10–30% headroom, or try your exact setup in our VRAM calculator.

Which GPU for LLM fine-tuning fits your model

• H100, 80 GB of HBM3. QLoRA on models up to 70B, where about 40 GB of 4-bit weights leaves room for activations at moderate context lengths. BF16 LoRA on 8B to 14B models. Full fine-tunes only for small models of around 3B parameters.
• H200, 141 GB of HBM3e. QLoRA on 70B with longer sequences or bigger batches, BF16 LoRA on 32B models with comfortable headroom, and full fine-tunes of 7B to 8B models with an 8-bit optimizer.
• B200, 180 GB of HBM3e. BF16 LoRA on 70B: about 140 GB of weights leaves roughly 40 GB for adapters and activations with gradient checkpointing. Also full fine-tunes of 8B models in standard mixed precision.
• B300, 288 GB of HBM3e. The largest single-GPU plan we offer, for 70B LoRA at long context without squeezing.

Memory bandwidth sets the pace once everything fits: 3.35 TB/s on the H100, 4.8 TB/s on the H200 and 8 TB/s on the B200, per NVIDIA's published specifications. For a closer look at the two Hopper cards, read our H100 vs H200 comparison.

When you need 8 GPUs

• Full fine-tunes above 8B. A 32B model needs roughly 510–640 GB of training state, which fits an HGX H200 node with 1.13 TB of HBM3e using FSDP or DeepSpeed ZeRO-3. A 70B model needs 1.1–1.4 TB: an HGX B300 node, or an HGX B200 node with optimizer offload to its 2 TB of RAM.
• Reinforcement learning. PPO and GRPO-style post-training often run a generation engine such as vLLM next to the trainer, so you split GPUs between rollouts and training.
• Long contexts. At 32k to 128k tokens, activations dominate the budget; context parallelism spreads them across GPUs.
• Wall-clock time. Data parallelism across eight GPUs shortens large runs that would take days on a single card.

Our 8-GPU servers connect all eight GPUs with NVLink. For pre-training and large post-training runs, see GPU servers for LLM training, and for a deeper look at the largest common size, read our guide to fine-tuning a 70B model.

Software and setup

Hugging Face TRL with PEFT, Axolotl, Unsloth and LLaMA-Factory all handle LoRA and QLoRA, with bitsandbytes for 4-bit loading and FSDP or DeepSpeed for multi-GPU full fine-tunes. Run them in Docker with the NVIDIA Container Toolkit, which is preinstalled on our Ubuntu images, so CUDA and PyTorch versions stay pinned from one experiment to the next. Tokenizing and packing a large dataset is CPU work, so the plan's CPU and RAM matter too: 24 vCPUs with 225 GB of RAM on the H100, 24 vCPUs with 240 GB on the H200, and 26 vCPUs with 256 GB on the B200.

Keep the base model and the tokenized dataset on local NVMe: 2 TB on the H100 and H200 plans, 2.75 TB on the B200. A 70B base model in BF16 is about 140 GB, so you download it once and reuse it all month, and inbound traffic does not count toward your bandwidth. After each run, load the checkpoint in vLLM on the same GPU and test it before you ship it; our PyTorch and vLLM guide covers the setup.

Monthly or hourly for fine-tuning?

Fine-tuning is iterative: clean the data, sweep learning rates, evaluate, repeat. A dedicated GPU for the month suits teams that run several experiments a week, because each extra run adds nothing to the bill. An H100 at $2,190 a month works out to about $3.00 per GPU-hour if it is busy around the clock; used for only 100 hours, the effective rate is about $21.90 per hour. If you fine-tune a few hours a quarter, hourly billing elsewhere will cost less. Run your own numbers against monthly vs hourly GPU pricing.

To start, send your base model, method, dataset size and context length through the contact form. We confirm the configuration, price and delivery date in writing, you pay by card or crypto, and your SSH credentials arrive by email.