How to choose a GPU server for AI

Knowing how to choose a GPU server for AI is mostly a matter of asking questions in the right order. Memory capacity decides what can run at all. Memory bandwidth and precision support decide how fast it runs. The interconnect and the CPU, RAM and storage around the GPUs decide whether you get the performance you paid for. Software and billing decide whether the server works for your team and your budget. This framework walks through each step, then maps common workloads to concrete configurations.

How to choose a GPU server for AI: memory first

Start by computing how much memory your workload needs, because it rules options in or out before anything else matters.

• Inference: weights (parameters × bytes per parameter) plus KV cache plus 10–30% overhead. A 70B model needs about 140 GB in BF16 and about 70 GB in FP8 before its cache.
• Full fine-tuning with Adam: roughly 16–20 bytes per parameter before activations, so about 1.1–1.4 TB for a 70B model.
• LoRA and QLoRA: far less, because the base weights are frozen. QLoRA on a 70B model fits on one 80 GB GPU.

Then decide whether the requirement fits on one GPU. Single GPUs in our catalog range from 32 GB (RTX 5090) through 48 GB (L40S), 80 GB (H100), 96 GB (RTX PRO 6000), 141 GB (H200) and 180 GB (B200) to 288 GB (B300). If one model must span several GPUs, look at the server total and at the interconnect, covered below. Our guide to VRAM for LLMs has the formulas and worked examples, and the VRAM calculator does the arithmetic for you.

Memory bandwidth and precision formats

Once the model fits, memory bandwidth largely sets inference speed. To generate each token, the GPU reads the model's active weights and the KV cache from memory, so at small batch sizes the time per token cannot be shorter than the bytes read divided by the bandwidth. Prompt processing and training are more compute-bound, so vendor FLOPS matter more there.

The ratios are what matter. An H200 has 43% more bandwidth than an H100, and a B200 about 2.4 times as much. The RTX PRO 6000 has more memory than an H100 but less than half its bandwidth, which suits it to models that fit comfortably, to many parallel copies and to rendering, more than to latency-critical serving of large models.

Precision support multiplies the effect. FP8 halves weight memory and bytes read per token compared with BF16, and FP4 halves them again. Hopper and Ada GPUs support FP8 in hardware; Blackwell GPUs and the MI355X add FP4. Lower precision costs some quality, more at 4 bits than at 8, so test on your own evaluation set. Our explainer on FP8 and FP4 precision goes into the formats.

Interconnect and network

If one model or one training job spans several GPUs, the link between them becomes part of the compute. Tensor parallelism exchanges activations at every layer, sharded training (FSDP or ZeRO-3) gathers weights at every step, and mixture-of-experts models shuffle tokens between experts. NVLink 4 gives each H100 or H200 900 GB/s and NVLink 5 gives each B200 or B300 1.8 TB/s, against roughly 128 GB/s in both directions combined for a PCIe Gen5 x16 slot: about 7 and 14 times as much. Our NVLink vs PCIe guide goes deeper.

PCIe is fine when GPUs work independently: one model copy per GPU behind a load balancer, separate experiments, or rendering. It is a poor fit for splitting a large model across cards.

Between servers, look at the network adapters. Our HGX servers carry one adapter per GPU: 8× 400G ConnectX-7 on H100, H200 and B200, and 8× 800G ConnectX-8 on B300. The MI355X server has 8× 400G RoCE, and the 8× RTX PRO 6000 server has 2× 100 GbE. If you plan multi-node training, ask about private networking between your servers. On the public side, single-GPU plans have 10 or 25 Gbps ports and include 20 TB of outbound transfer per month, and 8-GPU servers include 50 TB. At 10 Gbps, a 140 GB model downloads in about two minutes at line rate, while a 10 TB dataset takes over two hours, so plan large transfers ahead.

CPU, RAM and NVMe: keep the GPUs fed

A fast GPU waiting on a slow host is an expensive way to wait. Three host resources matter:

• CPU cores handle tokenization, data loading, image or video decoding and the inference server's scheduling. Vision and video pipelines need far more than text. Our 8-GPU servers have 112 to 128 cores, 14 to 16 per GPU; single-GPU plans have 16 to 28 vCPU.
• System RAM holds checkpoints while they load, datasets you cache and anything you offload. A common rule of thumb is at least as much RAM as total GPU memory, and more if you offload optimizer states to the CPU or merge large adapters there. Merging a LoRA adapter into a 70B model in BF16 takes about 140 GB of RAM: easy with the 225 GB of our single H100 plan, impossible on a 64 GB machine.
• Local NVMe stores weights, datasets and checkpoints. A resumable full fine-tuning checkpoint of a 70B model, with optimizer states, is over 1 TB. Our HGX servers carry 8× 3.84 TB of NVMe; single-GPU plans have 1 to 3.5 TB.

Software ecosystem: CUDA or ROCm

NVIDIA's CUDA stack has the broadest support: PyTorch, JAX, vLLM, SGLang and TensorRT-LLM all run on it, and most research code and custom kernels assume it. AMD's ROCm supports PyTorch, vLLM and SGLang, and the MI355X brings 288 GB per GPU; our 8× MI355X server costs $19,900 per month, about $3.41 per GPU-hour. The catch is the long tail: custom CUDA kernels, some quantization libraries and niche tools may need porting work or may not run at all. Before committing to AMD, run your exact stack end to end; our AMD ROCm documentation is the place to start.

Billing model: hourly, reserved or monthly

The same GPU can cost very different amounts depending on how you pay for it. On-demand billing by the second or hour suits short, unpredictable work. Spot or interruptible capacity is cheaper but can be reclaimed mid-run. Reserved terms lower the rate in exchange for a long commitment. Flat monthly dedicated servers suit steady workloads: training runs that last weeks, inference that serves around the clock, and teams that want a fixed budget.

To compare a monthly price with an hourly quote, divide one by the other: break-even hours = monthly price ÷ hourly rate. If you will use the GPU for more hours than that each month, monthly is cheaper. Our monthly vs hourly GPU pricing comparison goes deeper into the trade-offs. Availability matters too: in 2026, GPU capacity is tight and on-demand stock is often sold out, so a low rate only helps if the GPU is there when you need it. We bill monthly in advance, with a one-month minimum, no setup fee and cancellation any month, and we accept cards or crypto.

Workloads mapped to configurations

The table maps common AI workloads to configurations from our catalog. Treat it as a starting point and check your own numbers.

For frontier-scale training, rack-scale systems such as the GB300 NVL72 are available on request, from $7,990 per GPU per month on 12- to 36-month terms.

Checklist before you order

1. Write down the model, precision, maximum context length and expected concurrency, or, for training, the method, sequence length and micro-batch size.
2. Compute memory per GPU with 10–30% headroom, and decide between one GPU and several.
3. If one model spans GPUs, require NVLink, or accept the PCIe trade-off knowingly.
4. Confirm the precision formats your serving or training plan relies on, such as FP8 or FP4.
5. Check CPU cores, RAM (at least equal to GPU memory) and NVMe space for weights, datasets and checkpoints.
6. Estimate monthly outbound transfer against the included allowance, and plan off-server backups: we do not back up local disks unless agreed in writing.
7. Confirm the software stack: CUDA or ROCm, driver and CUDA versions, and the OS image (Ubuntu 22.04 or 24.04, with Debian or Rocky Linux on request).
8. Prepare your SSH public key and a firewall plan before the server arrives.
9. Choose the term: month to month, or 6 to 12 months with volume pricing if you need several servers.
10. Get the configuration, price, location and delivery date in writing before you pay. With us, if the confirmed delivery date is missed, you can cancel for a full refund.

When the list is done, compare configurations on our pricing page and send a request with your numbers. We confirm everything in writing before payment.

Frequently asked questions

What matters most when choosing a GPU server for AI?

Memory capacity comes first, because it decides whether your model, its KV cache or its training state fits at all. After that, compare memory bandwidth, precision support, interconnect and the CPU, RAM and storage around the GPUs.

Do I need NVLink?

You need it when one model or one training job is split across several GPUs, as with tensor parallelism or sharded training, because those exchange data at every layer or step. For independent jobs or one model copy per GPU, PCIe is fine.

How much system RAM should a GPU server have?

A common rule of thumb is at least as much RAM as total GPU memory, and more if you offload optimizer states to the CPU, merge large adapters or cache datasets in memory.

Should I choose NVIDIA or AMD for AI?

NVIDIA's CUDA stack has the broadest software support, while AMD's MI355X offers 288 GB per GPU on ROCm. Test your exact frameworks, kernels and quantization tools on ROCm before you commit.

Is monthly or hourly billing better for a GPU server?

Divide the monthly price by the hourly rate to get the break-even hours. Steady workloads that run most of the month are cheaper on a monthly plan, while short or occasional jobs are usually cheaper by the hour.