NCCL Tests for Multi-GPU Servers, Step by Step

Before a long training run, confirm that all eight GPUs talk to each other at full speed. These checks are quick and catch most interconnect problems early.

Running NCCL tests on a multi-GPU server is the quickest way to confirm that the GPUs, NVLink and the communication stack work together before you commit to a long training run. This page applies to our 8-GPU servers: check the topology, build NVIDIA's nccl-tests, run all_reduce_perf across eight GPUs, read busbw, then launch PyTorch jobs with torchrun. Multi-node notes come last. AMD servers use RCCL, covered in ROCm setup.

Check the GPU topology

nvidia-smi topo -m
nvidia-smi nvlink --status
systemctl status nvidia-fabricmanager

In the topo -m matrix, NV# means two GPUs are connected by NVLink, with # the number of links. PIX, PXB, PHB, NODE and SYS are PCIe paths, from closest to farthest. The matrix also shows each GPU's CPU and NUMA affinity.

On HGX servers (H100, H200, B200, B300), every GPU pair should show an NV entry and Fabric Manager should be active (running). The RTX PRO 6000 server has no NVLink: its GPUs talk over PCIe, so PCIe entries are expected there. Background in NVLink vs PCIe.

Build nccl-tests

Install the build tools and check for the NCCL development header:
```
sudo apt install -y git build-essential
ls /usr/include/nccl.h
```
If nccl.h is missing, install NCCL from NVIDIA's CUDA repository, set up as described in driver and CUDA versions. List the versions, then pick the newest one whose CUDA suffix (such as +cuda12.8) is not newer than the CUDA Version shown by nvidia-smi:
```
apt-cache madison libnccl2 | head
sudo apt install -y libnccl2=VERSION libnccl-dev=VERSION
```

Build the tests:

git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make -j"$(nproc)" CUDA_HOME=/usr/local/cuda

The binaries appear in ./build/.

To leave the host untouched, build the same way inside an NGC PyTorch container, which already includes CUDA and NCCL.

Run all_reduce_perf across 8 GPUs

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

One process drives eight GPUs (-g 8), from 8 bytes (-b) to 8 GB (-e), doubling the size at each step (-f 2). Each line reports the time and two bandwidth figures, algbw and busbw, for out-of-place and in-place runs. Check that:

the #wrong columns show 0 on every line, so the results are correct;
busbw rises with message size and levels off at the largest sizes;
the final "Avg bus bandwidth" line is stable from one run to the next.

Other collectives work the same way: all_gather_perf, reduce_scatter_perf and sendrecv_perf.

What busbw means

algbw is the data size divided by the time. It changes with the number of GPUs, so it is hard to compare across setups. busbw applies a correction factor so the figure reflects the traffic each link actually carries. For all-reduce, the factor is 2 × (n − 1) / n, which is 1.75 with eight GPUs. That makes busbw comparable with the hardware's link bandwidth and between servers of the same model.

Vendor link speeds are peak values that a test does not reach, so judge consistency instead: compare runs, and compare with an identical server if you have one. A sharp drop at large sizes, a result far below a twin server, or a hang usually points to a topology, Fabric Manager or hardware problem. Send us the full output and the topology matrix through the contact form.

Debug with NCCL_DEBUG=INFO

NCCL_DEBUG=INFO ./build/all_reduce_perf -b 1G -e 1G -g 8 2>&1 | tee nccl-debug.log

The log shows the NCCL version, the transport chosen between GPUs (P2P or NVLS over NVLink, SHM through host memory, NET over the network) and the rings and trees NCCL built. On HGX servers you should see NVLink transports. If you see SHM instead, check the topology and Fabric Manager. On the PCIe-based RTX PRO 6000 server, if the test hangs or reports errors, rerun it with NCCL_P2P_DISABLE=1. If that run works, peer-to-peer traffic is the problem, and we can help investigate.

Launch DDP or FSDP jobs with torchrun

torchrun starts one process per GPU and sets the variables that PyTorch's distributed package needs. In an environment with PyTorch, save this check as allreduce_check.py:

import os
import torch
import torch.distributed as dist

dist.init_process_group("nccl")
rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(rank)
x = torch.ones(1024, device="cuda") * (rank + 1)
dist.all_reduce(x)
print(f"rank {rank}: {x[0].item()}")
dist.destroy_process_group()

Run it on all eight GPUs:

torchrun --standalone --nproc_per_node=8 allreduce_check.py

Every rank should print 36.0, the sum of 1 to 8. Your DDP or FSDP training script starts the same way. DDP keeps a full copy of the model on each GPU and averages gradients. FSDP shards parameters, gradients and optimizer state across GPUs, so you can train models that do not fit in one GPU's memory. Put NCCL_DEBUG=INFO in front of the command when a job stalls.

Multi-node notes

Network. Ask for private networking between your servers and use the private addresses for rendezvous and NCCL traffic.
Interface. Find the private interface with ip -br addr and set NCCL_SOCKET_IFNAME to its name on every node, so NCCL does not pick the public interface.
Launch. Run the same command on each node, changing only the rank. Here 10.0.0.1 stands for the first node's private address:
```
torchrun --nnodes=2 --nproc_per_node=8 --node_rank=0 \
  --master_addr=10.0.0.1 --master_port=29500 train.py
```
Use --node_rank=1 on the second node, and allow the port between the nodes if a firewall is active.
Fabric. Full-speed GPU traffic between nodes needs an RDMA fabric (InfiniBand or RoCE) cabled and configured between your servers. That is part of a GPU cluster build: contact us to plan it.