ROCm setup for AMD Instinct MI355X

This ROCm setup for AMD Instinct MI355X servers takes you from the first login to PyTorch, vLLM and multi-GPU jobs. Our 8-GPU MI355X servers are bare metal and ship with a ROCm image, so the kernel driver and ROCm are already installed. Each GPU has 288 GB of HBM3E, per AMD's published specifications. Prefer containers and wheels built on ROCm 7.x: builds made for older ROCm releases may not include code for the MI355X, whose architecture name is gfx950.

Check the GPUs with rocm-smi, amd-smi and rocminfo

rocm-smi
amd-smi list
amd-smi static
rocminfo | grep -E 'Marketing Name|gfx'

You should see eight GPUs, numbered 0 to 7, in rocm-smi and amd-smi list. rocminfo lists the CPUs and every GPU as agents, and each GPU should report gfx950. amd-smi is AMD's newer tool: it covers what rocm-smi does, and amd-smi static adds detailed information about each GPU. For ongoing checks, see GPU monitoring.

Add users to the render and video groups

Root can use the GPUs directly. Other users need access to /dev/kfd and /dev/dri, which ROCm grants through the render and video groups. For each non-root user (here bob):

sudo usermod -aG render,video bob
# after bob logs out and back in
groups bob

The output should include render and video. Without them, ROCm programs find no GPUs or fail with permission errors on /dev/kfd.

Run PyTorch with ROCm containers or wheels

The simplest path is AMD's PyTorch container, which bundles a matching ROCm user space. If Docker is not on your image, install it first (on Ubuntu, sudo apt install -y docker.io). The device flags are explained in Docker GPU containers.

docker run -it --rm \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined \
  rocm/pytorch:latest

Inside the container, check what PyTorch sees:

python3 -c "import torch; print(torch.version.hip, torch.cuda.device_count(), torch.cuda.get_device_name(0))"

You should see a HIP version, 8 and the GPU name. PyTorch on ROCm keeps the torch.cuda API, so most CUDA code runs unchanged. For reproducible builds, pin a specific tag from the rocm/pytorch repository instead of latest.

To install on the host instead, use a virtual environment and the ROCm wheel index that PyTorch's install selector lists for your ROCm version:

sudo apt install -y python3-venv
python3 -m venv ~/venvs/rocm
source ~/venvs/rocm/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/rocmX.Y

Replace rocmX.Y with that version, and check that the PyTorch release supports the MI355X. If it does not, you will see hipErrorNoBinaryForGpu.

Choose which GPUs a process sees

HIP_VISIBLE_DEVICES=0,1 python3 train.py

# the same inside a container
docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \
  -e HIP_VISIBLE_DEVICES=0,1 rocm/pytorch:latest \
  python3 -c "import torch; print(torch.cuda.device_count())"

The container command should print 2. Visible GPUs are renumbered from 0 inside the process. HIP also honors CUDA_VISIBLE_DEVICES, and ROCR_VISIBLE_DEVICES filters at a lower level; set only one of them to avoid confusing combinations. To run two jobs side by side, give each its own set, such as GPUs 0 to 3 and 4 to 7.

Serve models with vLLM in a ROCm container

AMD publishes ROCm-enabled vLLM images. Start one with the same device flags, publish the port on the loopback interface only, and keep model weights on your data drive:

docker run -d --name vllm \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined \
  -p 127.0.0.1:8000:8000 \
  -v /data/hf:/data/hf -e HF_HOME=/data/hf \
  rocm/vllm:latest \
  vllm serve Qwen/Qwen2.5-72B-Instruct --tensor-parallel-size 8 --host 0.0.0.0 --port 8000

Follow startup with docker logs -f vllm, then run curl http://127.0.0.1:8000/v1/models: you should see the model listed. Inside the container vLLM listens on all interfaces, while the 127.0.0.1 mapping keeps it private on the host. A 72B model needs roughly 145 GB for its weights in BF16, so it fits on one MI355X; spreading it over eight GPUs leaves far more memory for the KV cache. Options, gated models and API testing are covered in serving an LLM with vLLM, and exposing the endpoint safely in the firewall guide.

Multi-GPU communication with RCCL

RCCL is ROCm's collective communication library and implements the NCCL API. PyTorch on ROCm uses it when you select the nccl backend, so DDP and FSDP scripts launched with torchrun --standalone --nproc_per_node=8 run unchanged, and RCCL reads the usual NCCL variables, such as NCCL_DEBUG=INFO. To measure GPU-to-GPU bandwidth, build AMD's rccl-tests, which mirror NVIDIA's nccl-tests:

sudo apt install -y git build-essential
git clone https://github.com/ROCm/rccl-tests.git
cd rccl-tests
make
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

The #wrong columns should show 0 on every line. Reading the bandwidth columns is explained in NCCL tests. For multi-node jobs over the servers' RoCE network, contact us to plan the fabric.

Troubleshooting

• hipErrorNoBinaryForGpu. The build contains no code for gfx950. Use a ROCm 7.x-based image or a wheel that supports the MI355X.
• Permission denied on /dev/kfd, or no GPUs found. Add the user to the render and video groups and log in again. In Docker, pass both --device flags.
• Fewer than eight GPUs listed. Run sudo dmesg | grep -i amdgpu and send the output, with amd-smi list, through the contact form.