GPU servers for LLM inference, running 24/7 at a flat price

GPU servers for LLM inference pay off when they run all day: a model that serves traffic 24/7 uses every hour of a monthly plan. We rent whole NVIDIA GPUs by the month with root access, so you run vLLM, SGLang or TensorRT-LLM exactly as you would on your own hardware. This page covers the sizing that decides your bill: weights, KV cache, precision and batching.

Size the weights first, then the KV cache

Weights take roughly parameters × bytes per parameter: 2 in BF16, 1 in FP8 and about 0.5 in FP4, plus a little for quantization scales. The KV cache comes next and usually limits concurrency. Per token, it takes 2 (keys and values) × layers × KV heads × head dimension × bytes per value.

For a Llama-style 70B model with 80 layers, 8 KV heads and a head dimension of 128, that is about 0.33 MB per token in BF16, and half that in FP8. One 128k-token context needs about 43 GB of cache in BF16; 32 concurrent requests of 4,000 tokens need about 42 GB.

The runtime takes its share too: vLLM uses up to 90% of GPU memory by default for weights, activations and cache, and CUDA graphs need some room of their own. To size other models, use our VRAM calculator.

Choosing GPU servers for LLM inference

Generating tokens is mostly limited by memory bandwidth, because each decode step reads the active weights once. NVIDIA lists 3.35 TB/s for the H100, 4.8 TB/s for the H200, 8 TB/s for the B200 and 1.6 TB/s for the RTX PRO 6000. Prefill, the processing of the prompt, is compute-bound and benefits from FP8 and FP4 tensor cores.

• 8B to 14B models: an RTX PRO 6000 holds the weights in BF16 with most of its 96 GB left for cache.
• 32B models: FP8 weights of about 32 GB fit an RTX PRO 6000 or an H100 with room for concurrency.
• 70B models: an H200 in FP8, a B200 in FP8 or FP4, or an RTX PRO 6000 in FP4.
• Large mixture-of-experts models: DeepSeek-V3-class models have about 671 billion parameters, roughly 700 GB in their native FP8, so they need an HGX H200 node (1.13 TB) or an HGX B200 node (1.44 TB); an HGX H100 node has 640 GB in total. See our DeepSeek hardware requirements.

FP8 runs natively on Hopper and Blackwell, while FP4 on NVIDIA hardware needs Blackwell: the B200, B300, RTX PRO 6000 or RTX 5090. Quality after quantization depends on the model and the method, so compare outputs on your own evaluation set; background in FP8 and FP4 precision explained. On HGX nodes, NVLink makes tensor parallelism across GPUs efficient. The RTX PRO 6000 has no NVLink, so on the 8-GPU RTX PRO server, run one replica per GPU, or small tensor-parallel groups, behind your load balancer.

Serving stack and batching

vLLM, SGLang and TensorRT-LLM all batch requests continuously and manage the KV cache in pages, so throughput rises with concurrency until the cache fills or the GPU saturates. Prefix caching helps when many requests share a long system prompt. With vLLM in Docker, a typical start looks like this:

docker run --gpus all --ipc=host -p 8000:8000 \
  -v /data/hf:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model your-org/your-model-fp8 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY"

Cap --max-model-len at what your product needs, since the longest context sets the worst-case cache per request, and pin an image version in production. The server speaks an OpenAI-compatible protocol; keep its port behind a firewall or a TLS reverse proxy, as in our networking and firewall guide, and follow the PyTorch and vLLM guide for the full setup.

Why a flat monthly price suits 24/7 serving

Over 730 hours a month, an H200 at $3,090 works out to about $4.23 per GPU-hour, a B200 at $4,590 to about $6.29 and an RTX PRO 6000 at $1,390 to about $1.90. Those rates hold only when the GPU is busy: an H200 that serves traffic 8 hours a day effectively costs about $12.70 per used hour. Steady or growing traffic suits a monthly plan. Spiky traffic with long idle stretches, or a prototype with a handful of users, may cost less on hourly or per-token billing elsewhere; see monthly vs hourly GPU pricing.

Bandwidth is included: 20 TB per month on single-GPU plans and 50 TB on 8-GPU servers, under fair use, with no egress fees and inbound traffic not counted. Text is light: even at a few hundred bytes per streamed token, 20 TB covers tens of billions of generated tokens.

Getting started

Tell us the model, precision, context length and expected concurrency through the contact form. We confirm the configuration, price, location and delivery date in writing, you pay by card or crypto, and SSH credentials arrive by email. Our Ubuntu images ship with NVIDIA drivers, CUDA, Docker and the NVIDIA Container Toolkit, so the container above runs once the weights are downloaded. To scale out, add servers and link them with private networking on request.