GPU server firewall setup

A GPU server firewall setup has one goal: only the ports you mean to open are reachable. Every FullGPU server has a public IPv4 address and an IPv6 /64, so anything listening on all interfaces is exposed unless you filter it. This page sets up ufw, explains how Docker bypasses it, then shows how to secure a vLLM endpoint with a reverse proxy, TLS and an API key.

Set up ufw, allowing SSH first

Allow SSH before you enable the firewall, or you will lock yourself out:

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw enable
sudo ufw status verbose

Answer y when ufw warns that the command may disrupt SSH connections. You should see Status: active, the default policies, and a rule allowing 22/tcp (OpenSSH) for IPv4 and IPv6. Keep your session open and test a new SSH connection from a second terminal.

With a fixed office or VPN address, you can restrict SSH to it. Add the new rule before deleting the old one:

sudo ufw allow from 198.51.100.7 to any port 22 proto tcp
sudo ufw delete allow OpenSSH

A softer option is sudo ufw limit OpenSSH, which rate-limits repeated connections from one address. Key-only login, covered in SSH access, matters more than either.

Filter IPv6 too

grep IPV6 /etc/default/ufw
sudo ss -tulpn

ufw filters IPv6 when /etc/default/ufw contains IPV6=yes, the Ubuntu default. With IPV6=no, ufw leaves IPv6 unfiltered and services listening on [::] stay reachable through your /64. Set it to yes, run sudo ufw disable && sudo ufw enable, then add your rules again so they get IPv6 copies. ss -tulpn lists every listening socket with its process: anything on 0.0.0.0, * or [::] listens on all interfaces. Make sure you need it there and that the firewall covers it, or bind it to 127.0.0.1.

Docker bypasses ufw for published ports

Docker manages its own iptables rules. When you publish a port with -p 8000:8000, Docker redirects that traffic to the container through its own rules, and it never passes the ufw rules that protect the host. The port is open to the internet even with default deny incoming, which catches many vLLM, Jupyter and monitoring setups.

1. Publish on the loopback interface only: -p 127.0.0.1:8000:8000, or "127.0.0.1:8000:8000" in Compose.
2. Check the binding with sudo ss -tlnp | grep 8000. It should show 127.0.0.1:8000.
3. From your laptop, curl http://203.0.113.10:8000/v1/models (with your server's IP) should fail to connect.
4. Expose the service through a reverse proxy on the host, which ufw does filter.

To make loopback the default for every published port, add "ip": "127.0.0.1" to /etc/docker/daemon.json, keeping existing entries such as the NVIDIA runtime, then restart Docker. For finer control, Docker evaluates the DOCKER-USER iptables chain before its own rules.

Put a reverse proxy with TLS in front

Point a DNS name (here llm.example.com) at your server's addresses and open the web ports, then pick Caddy or nginx. Both configurations below expose only the /v1/ API paths.

sudo ufw allow 80,443/tcp

Caddy

Caddy obtains and renews certificates automatically. It is packaged in Ubuntu 24.04 (sudo apt install -y caddy); on 22.04, install it from Caddy's own repository. Replace the contents of /etc/caddy/Caddyfile with:

llm.example.com {
    handle /v1/* {
        reverse_proxy 127.0.0.1:8000
    }
    handle {
        respond 404
    }
}

Apply it with sudo systemctl reload caddy. Once the certificate is issued, https://llm.example.com/v1/models answers over HTTPS.

nginx

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/llm:

server {
    listen 80;
    server_name llm.example.com;

    location /v1/ {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }

    location / {
        return 404;
    }
}

Enable it, then request a certificate. certbot adds the HTTPS configuration and sets up renewal:

sudo ln -s /etc/nginx/sites-available/llm /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d llm.example.com

proxy_buffering off lets streamed tokens reach clients as they are generated, and the longer timeout covers long generations.

Require an API key on OpenAI-compatible servers

TLS protects traffic in transit; an API key decides who may use the model. vLLM requires a bearer token when you start it with --api-key or set VLLM_API_KEY:

openssl rand -hex 32    # generate a key
vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000 --api-key YOUR_KEY

Test from your laptop:

curl https://llm.example.com/v1/models -H "Authorization: Bearer YOUR_KEY"
curl -i https://llm.example.com/v1/models

The first command should list the model and the second should return status 401. vLLM applies the key to the /v1 routes, and other routes such as /metrics may stay open, which is why the proxy exposes nothing else. Prefer the environment variable, as in the systemd setup from serving an LLM with vLLM: a key typed on the command line is visible to other local users in the process list. Other OpenAI-compatible servers have similar options.

Private networking between your servers

On request, we connect your servers on a private network. Use it for traffic between them: data sync, NCCL and torchrun rendezvous, internal APIs and metrics. Allow it in ufw by subnet, or by interface (find it with ip -br addr):

sudo ufw allow from 10.0.0.0/24
# or
sudo ufw allow in on PRIVATE_INTERFACE

Replace the subnet or interface with those of your private network. See private networking to request it.

DDoS mitigation is included

Every public IP address on your servers has DDoS mitigation, included in the price. It protects against network-level attacks; it does not replace a firewall, TLS, authentication or rate limits on your own services. If you see unusual traffic or think your server is under attack, contact us. More on DDoS protection.

Troubleshooting

• Locked out after enabling ufw. Do not keep retrying. Contact support with the server IP and the rules you added.
• Caddy or certbot cannot get a certificate. Check that the DNS name points to the server, that ports 80 and 443 are allowed in ufw, and that nothing else listens on them (sudo ss -tlnp).
• A container port is reachable despite ufw. It is published on all interfaces. Recreate the container with -p 127.0.0.1:PORT:PORT.