Serve an LLM with vLLM

This guide shows how to serve an LLM with vLLM on a FullGPU server: create a Python environment, install PyTorch and vLLM, start an OpenAI-compatible endpoint on one GPU or across eight, test it with curl and run it as a systemd service. Before you pick a model, estimate its memory with the VRAM calculator. In BF16, weights take roughly 2 bytes per parameter, plus room for the KV cache.

Install PyTorch and vLLM in a Python environment

Create the environment

uv creates environments and installs packages quickly, and can fetch a Python version your OS does not ship:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv ~/venvs/vllm --python 3.12
source ~/venvs/vllm/bin/activate

With the standard tools instead, run sudo apt install -y python3-venv, then python3 -m venv ~/venvs/vllm and the same source line. Keep vLLM in its own environment, separate from your training code.

Install the packages

uv pip install vllm --torch-backend=auto
python -c "import torch, vllm; print(torch.__version__, torch.cuda.is_available(), vllm.__version__)"

You should see True between the two versions. vLLM pins the PyTorch version it was built against and installs it for you, and --torch-backend=auto lets uv pick the PyTorch build that matches your driver. Do not install another PyTorch version on top. For a training environment without vLLM, install PyTorch alone the same way with uv pip install torch --torch-backend=auto. With plain pip, run pip install vllm.

Start the OpenAI-compatible server

On a single GPU, a 7B-class model is a good first test:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

On an 8-GPU server, split a larger model across all GPUs with tensor parallelism:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 32768

The first start downloads the weights, which can take a while. The server is ready when the log shows Application startup complete. Weights go to ~/.cache/huggingface; to keep them on your NVMe data drive, run export HF_HOME=/data/hf before starting.

Test the API with curl

curl http://127.0.0.1:8000/v1/models

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 32}'

The first call should list the model. The second returns JSON with the answer in choices[0].message.content. Any OpenAI-compatible client works with its base URL set to http://127.0.0.1:8000/v1. From your laptop, reach it through an SSH tunnel: ssh -N -L 8000:127.0.0.1:8000 gpu1.

Use a Hugging Face token for gated models

Some models, such as Meta's Llama family, require you to accept a license on Hugging Face. Accept it on the model page with your account, create an access token with read permission, then pass it to vLLM in the HF_TOKEN variable. Typing it at a prompt keeps it out of your shell history:

read -rsp 'Hugging Face token: ' HF_TOKEN && export HF_TOKEN
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 8000 --max-model-len 16384

A 401 or 403 error during the download means the token is missing or wrong, or the license was not accepted for that account.

Run vLLM as a systemd service

1. Store settings and secrets in a file only root can read. Generate the API key with openssl rand -hex 32; vLLM reads VLLM_API_KEY as if you had passed --api-key.

   sudo mkdir -p /etc/vllm
   sudo tee /etc/vllm/vllm.env >/dev/null <<'EOF'
   HF_HOME=/data/hf
   HF_TOKEN=hf_your_token_here
   VLLM_API_KEY=replace-with-a-long-random-string
   EOF
   sudo chmod 600 /etc/vllm/vllm.env

2. Create /etc/systemd/system/vllm.service. This example runs as root with the environment created above; adjust User and the paths for another account:

   [Unit]
   Description=vLLM OpenAI-compatible server
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   User=root
   EnvironmentFile=/etc/vllm/vllm.env
   ExecStart=/root/venvs/vllm/bin/vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000 --max-model-len 16384
   Restart=on-failure
   RestartSec=10
   
   [Install]
   WantedBy=multi-user.target

3. Start it and follow the log:

   sudo systemctl daemon-reload
   sudo systemctl enable --now vllm
   sudo journalctl -u vllm -f

   When the log shows Application startup complete, the curl tests work again once you add the header -H "Authorization: Bearer YOUR_KEY". Without it, you should get 401 Unauthorized. The service restarts after a crash and starts at boot.

Alternative: the vllm/vllm-openai image

docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v /data/hf:/data/hf \
  --env-file /etc/vllm/vllm.env \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct --max-model-len 16384

The image's entrypoint is the vLLM server, so arguments after the image name go straight to it. The env file sets HF_HOME=/data/hf, which is why that path is mounted. Pin a version tag instead of latest in production. On AMD servers, use the ROCm image from ROCm setup.

Secure the endpoint

For personal use, keep vLLM on 127.0.0.1 and use the SSH tunnel. To serve applications over the internet, put a reverse proxy with TLS in front, require an API key and expose only the /v1/ paths: the key protects the /v1 routes, and other routes such as /metrics may stay open. The firewall guide covers ufw, Docker's published ports and a Caddy or nginx setup. For sizing and running costs, read self-hosting LLM inference with vLLM.

Troubleshooting

• CUDA out of memory at startup. Lower --max-model-len, spread the model over more GPUs with --tensor-parallel-size, or choose a smaller or quantized model. Check with nvidia-smi that no other process holds GPU memory.
• Attention heads not divisible by the tensor parallel size. Pick a size that divides the model's head count, such as 4 instead of 8.
• 401 Unauthorized. The Authorization header is missing or the key does not match.