How to self-host LLM inference with vLLM

To self-host LLM inference with vLLM, you need three things: a GPU with enough memory for the weights and the KV cache, a vLLM server that exposes an OpenAI-compatible API, and the usual production hygiene around it, meaning authentication, TLS, a firewall, a service manager and monitoring. This guide covers each step with commands you can adapt, using Llama 3.1 models as examples, on an Ubuntu server with NVIDIA drivers already installed.

Why self-host LLM inference with vLLM

vLLM is an open-source inference engine built around a paged KV cache and continuous batching, which let one GPU serve many requests at once. Running it yourself, rather than calling a hosted API, gives you four things:

• Data control. Prompts and outputs stay on a machine you control, with no third-party retention policy to evaluate.
• Flat cost. Per-token pricing grows with usage. A dedicated GPU costs the same whether it serves a million tokens or a billion, so your cost per token is the monthly price divided by the tokens you serve, and it falls as utilization rises.
• Latency you control. No shared queue, a server placed close to your application, and your own trade-off between batch size and response time.
• Customization. Any open-weight model, your own fine-tunes and LoRA adapters, structured outputs, and context limits that suit your application.

The trade-offs are real. You operate the stack: upgrades, security, capacity and failover are yours. You pay for idle hours. Closed frontier models are not available to self-host. And if your volume is low or spiky, a per-token API is often cheaper.

Size the GPU: weights plus KV cache

Memory decides what you can serve. Weights take parameters × bytes per parameter: about 16 GB for Llama 3.1 8B in BF16, and about 70 GB for Llama 3.1 70B in FP8 or 140 GB in BF16. The KV cache takes 2 × layers × KV heads × head dimension × bytes per value for every token in flight: about 0.13 MB per token for the 8B model and 0.33 MB for the 70B, in BF16.

A worked example: Llama 3.1 70B in FP8 on one 141 GB H200, with --gpu-memory-utilization 0.92. vLLM may use about 130 GB. After roughly 70 GB of weights and a few GB for activations, about 55 GB remain for the cache: some 170,000 tokens in BF16, or 20 concurrent conversations at a full 8,192 tokens each. An FP8 cache doubles that to about 40. Our guide to VRAM for LLMs has the full method and more examples.

Install vLLM and start an OpenAI-compatible server

Install vLLM in a virtual environment. Its wheels bring their own PyTorch build, so the system only needs a recent NVIDIA driver; check it with nvidia-smi. See also our PyTorch and vLLM documentation.

sudo apt-get update
sudo apt-get install -y python3-venv
python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install vllm

Llama models are gated on Hugging Face: accept the license on the model page, create an access token and export it. Then generate an API key and start the server, bound to localhost:

export HF_TOKEN=hf_your_token
export VLLM_API_KEY=$(openssl rand -hex 32)

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --api-key "$VLLM_API_KEY" \
  --max-model-len 32768

The first start downloads the weights into the Hugging Face cache; set HF_HOME to a directory on the local NVMe if your home directory is small. Then test the endpoint:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain the KV cache in one sentence."}],
    "max_tokens": 100
  }'

Because the API follows OpenAI's format, existing clients work once you change the base URL and the key:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="YOUR_KEY")
reply = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

If you prefer Docker, the official vllm/vllm-openai image takes the same arguments. Publish the port on 127.0.0.1 only, and in production pin a release tag instead of latest:

docker run --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --api-key "$VLLM_API_KEY"

Tensor parallelism across GPUs

When a model does not fit on one GPU, --tensor-parallel-size splits every layer across several. The weights and the KV cache are divided between the GPUs, which exchange activations at every layer, so NVLink matters here. The size must divide the model's number of attention heads: Llama 3.1 70B has 64, so use 2, 4 or 8.

On an 8× H100 server, you can run two copies of Llama 3.1 70B in BF16, each on four GPUs:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 --host 127.0.0.1 --port 8001

CUDA_VISIBLE_DEVICES=4,5,6,7 vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 --host 127.0.0.1 --port 8002

Each copy spreads 140 GB of weights over 320 GB, about 35 GB per GPU, which leaves roughly as much again on each GPU for the cache at the default memory setting. A reverse proxy balances requests across the two ports.

The rule of thumb: use tensor parallelism when the model does not fit on one GPU or when you need lower latency per request. Otherwise, run one copy per GPU behind a load balancer, which avoids inter-GPU traffic entirely. On PCIe-only cards such as the L40S or RTX PRO 6000, prefer separate copies, or try --pipeline-parallel-size, which communicates less than tensor parallelism.

Tune memory: FP8, prefix caching and limits

A handful of options decide how much fits and how well the cache is used:

• --quantization fp8 converts BF16 weights to FP8 as the model loads, halving weight memory on Hopper, Ada and Blackwell GPUs. Check how your vLLM version does this: older versions loaded the full-precision weights first, so the BF16 model had to fit before conversion. For a 70B model on one GPU, use a checkpoint that is already quantized to FP8; vLLM detects the format from the checkpoint's configuration.
• --kv-cache-dtype fp8 halves the cache and doubles the tokens you can hold. Check quality on long-context tasks before you adopt it.
• Prefix caching reuses the cache for shared prompt prefixes: system prompts, few-shot examples, earlier turns of a chat and documents repeated across RAG queries. Recent versions enable it by default; on older ones, pass --enable-prefix-caching. Put stable content first in your prompts and variable content last to benefit from it.
• --max-model-len caps the context per request. The default is the model's maximum, 131,072 tokens for Llama 3.1, and vLLM refuses to start if one sequence of that length cannot fit in the cache, so set it to what you actually need.
• --gpu-memory-utilization is the fraction of each GPU's memory that vLLM may use for weights, activations and cache, 0.9 by default. On a dedicated GPU you can raise it to about 0.95 for more cache; lower it if anything else shares the GPU.
• --max-num-seqs caps how many sequences run at once, a simple way to bound latency under load.

Put together, serving a pre-quantized 70B FP8 checkpoint on one H200 looks like this:

vllm serve /models/llama-3.1-70b-instruct-fp8 \
  --served-model-name llama-3.1-70b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --host 127.0.0.1 --port 8000

Our explainer on FP8 and FP4 precision covers what quantization costs in quality.

Secure the endpoint

An open inference endpoint gets found and used by others. Lock it down in layers.

Bind to localhost and require a key

The examples above bind vLLM to 127.0.0.1, so nothing outside the server can reach it directly. --api-key, or the VLLM_API_KEY environment variable, makes clients send the key in an Authorization: Bearer header. The key guards the /v1 API routes; do not assume it covers every other route the server exposes, such as metrics, which is one more reason to keep the port private. If several teams or customers share the endpoint, put an API gateway in front for per-user keys and rate limits.

Firewall

Allow SSH, HTTP and HTTPS, and deny everything else:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

If only known clients should connect, replace the 443 rule with one rule per source address, such as sudo ufw allow from 203.0.113.10 to any port 443 proto tcp. One trap: ports published by Docker bypass UFW rules, which is why the Docker example publishes on 127.0.0.1 only. Our networking and firewall guide has more detail. Our servers also include DDoS mitigation on public IPs, which complements authentication but does not replace it.

Reverse proxy with TLS

Caddy obtains and renews a certificate automatically once your DNS record points at the server and ports 80 and 443 are open. After you install it from its official package repository, the whole configuration in /etc/caddy/Caddyfile is:

llm.example.com {
    reverse_proxy 127.0.0.1:8000
}

Reload it with sudo systemctl reload caddy. Streamed responses pass through as they are generated. To balance two vLLM copies, list both upstreams: reverse_proxy 127.0.0.1:8001 127.0.0.1:8002. If you use nginx instead, turn off proxy_buffering for the API location and raise proxy_read_timeout, or streamed tokens will arrive in bursts and long requests will time out.

Run vLLM as a service and monitor it

In production, run vLLM under systemd as an unprivileged user, so that it starts at boot and restarts after a crash:

sudo useradd --system --user-group --create-home --home-dir /opt/vllm --shell /usr/sbin/nologin vllm
sudo -u vllm python3 -m venv /opt/vllm/venv
sudo -u vllm /opt/vllm/venv/bin/pip install vllm
sudo mkdir -p /etc/vllm

Keep secrets in /etc/vllm/vllm.env, readable by root only (sudo chmod 600 /etc/vllm/vllm.env); systemd reads the file before it drops privileges:

HF_TOKEN=hf_your_token
HF_HOME=/opt/vllm/hf-cache
VLLM_API_KEY=your-long-random-key

Then create /etc/systemd/system/vllm.service:

[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target

[Service]
User=vllm
Group=vllm
EnvironmentFile=/etc/vllm/vllm.env
ExecStart=/opt/vllm/venv/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 8000 --max-model-len 32768
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
journalctl -u vllm -f

Here vLLM reads the key from VLLM_API_KEY, so it never appears on the command line. For monitoring, watch three layers:

• Health. GET /health returns 200 when the server is up; point an uptime check at it.
• Serving metrics. /metrics exposes Prometheus metrics, including running and waiting requests (vllm:num_requests_running and vllm:num_requests_waiting), time to first token, end-to-end latency and KV cache usage. Alert when requests wait for minutes, when time to first token climbs, or when the cache stays near full, which forces vLLM to preempt requests.
• GPU metrics. Use nvidia-smi for a quick look and NVIDIA's DCGM exporter for Prometheus. Expect memory to show about 90% used even when idle, because vLLM reserves its share at startup. Watch temperature, power and GPU errors in the kernel log (journalctl -k).

For more on the GPU side, see our GPU monitoring guide.

Choose the GPU

Match the GPU to the model, the precision and the concurrency you need. For generation, memory bandwidth largely sets speed, so it is the second number to compare after capacity.

A 70B model in FP8 also fits on two H100s with tensor parallelism, but at $4,380 per month that costs more than one H200 and adds inter-GPU traffic. vLLM also supports AMD GPUs through ROCm, so the 8× MI355X server, with 288 GB per GPU, is an option for very large models; test your model on it first. With a flat monthly price, the cost of a self-hosted endpoint is known in advance, and our plans include 20 TB (single GPU) or 50 TB (8 GPUs) of outbound transfer per month, far more than streamed text usually needs. For a deeper comparison, see our guide to the best GPU for LLM inference or the NVIDIA H200 page.

Frequently asked questions

What GPU do I need to self-host a 70B model with vLLM?

In FP8 the weights take about 70 GB, so a single 141 GB H200 serves it with room for concurrent requests, while an 80 GB H100 fits it only tightly. In BF16 the weights alone are about 140 GB, which points to a B200, a B300 or tensor parallelism across several GPUs.

Is vLLM compatible with the OpenAI API?

Yes. vLLM exposes OpenAI-style endpoints such as /v1/chat/completions, /v1/completions and /v1/models, so most OpenAI client libraries work once you change the base URL and the API key.

How do I secure a vLLM endpoint?

Bind vLLM to 127.0.0.1, require an API key, and expose it only through a reverse proxy with TLS, behind a firewall that allows only SSH, HTTP and HTTPS. In Docker, publish the port on 127.0.0.1, because ports published by Docker bypass UFW.

Why does nvidia-smi show the GPU almost full when vLLM is idle?

vLLM reserves a fixed share of GPU memory at startup, 90% by default, and uses most of it for KV cache blocks. That is expected behavior, not a memory leak.

Should I use tensor parallelism or several replicas?

Use tensor parallelism when the model does not fit on one GPU or when you need lower latency per request, ideally over NVLink. If the model fits on one GPU, separate replicas behind a load balancer avoid inter-GPU traffic and usually give more total throughput.