NVLink vs PCIe: bandwidth, topology and when it matters

NVLink vs PCIe is a question of how fast GPUs can exchange data with each other. PCIe is the general-purpose bus that connects a GPU to the rest of the server. NVLink is NVIDIA's dedicated GPU-to-GPU link, with roughly 7 to 14 times the bandwidth of a PCIe Gen5 x16 slot per GPU. Whether you need it depends on how much your GPUs talk to each other: for tensor parallelism and large-scale training it is decisive; for single-GPU work and data-parallel serving it barely matters. Here are the numbers, and the workloads on each side.

NVLink vs PCIe: the bandwidth numbers

One trap first: PCIe figures are often quoted per direction, while NVIDIA quotes NVLink as the total of both directions. Compare like with like. Figures are per GPU for PCIe and NVLink, and per adapter for network links.

So NVLink 4 gives each GPU about 7 times the bandwidth of a PCIe Gen5 x16 slot, and NVLink 5 about 14 times. The PCIe figures follow from the PCIe specification (32 GT/s per lane for Gen5 and 16 GT/s for Gen4, after encoding overhead) and are peaks; real transfers land somewhat lower. The NVLink figures are NVIDIA's.

NVSwitch, HGX boards and NVL72 racks

Bandwidth per GPU is half the story. Topology, meaning which GPU can reach which other GPU at full speed, is the other half.

• PCIe servers. GPUs sit on CPU root ports or PCIe switches. GPU-to-GPU traffic crosses those switches and, between CPU sockets, the link between the CPUs. It may also share bandwidth with network adapters and NVMe drives.
• HGX boards. Eight SXM GPUs sit on one baseboard with NVSwitch chips that connect every GPU to every other at full NVLink bandwidth. All eight can exchange data at once, in any pattern, which is exactly what collective operations need.
• NVL72 racks. The GB300 NVL72 extends the NVLink domain from 8 GPUs to 72 across a liquid-cooled rack, with 130 TB/s of total NVLink bandwidth according to NVIDIA. Collectives across all 72 GPUs never touch the network.
• AMD MI355X. AMD's 8-GPU platform takes a third approach: no switch, but seven Infinity Fabric links per GPU, at 153.6 GB/s each per AMD, one to each of the other seven GPUs.

When the interconnect matters

Three patterns make GPU-to-GPU bandwidth the bottleneck.

Tensor parallelism

Tensor parallelism splits every layer across GPUs, and the standard scheme needs two all-reduce operations per transformer layer in every forward pass. Take a 70B-class model (80 layers, hidden size 8,192) served with tensor parallelism across 8 GPUs, processing an 8K-token prompt in BF16. Each all-reduce covers 8,192 tokens × 8,192 values × 2 bytes = 128 MiB. With the standard ring estimate, each GPU sends 2 × 7/8 of that per all-reduce, 160 times: about 37.6 GB per GPU for one prefill. At ideal per-direction bandwidth:

These are lower bounds that ignore latency and any overlap with compute, but the ratio is what matters. During decode the messages are tiny, so latency rather than bandwidth dominates, and 160 small all-reduces per decode step add up on PCIe.

Large all-reduce in training

Data-parallel training synchronizes gradients at every step. A 7B model has about 14 GB of gradients in BF16, and a ring all-reduce over 8 GPUs moves about 24.5 GB per GPU: roughly 0.38 s at PCIe Gen5's per-direction rate against about 0.05 s on NVLink 4, at every step. Frameworks overlap this with the backward pass, so it only hurts when communication outlasts computation, but sharded approaches such as FSDP and ZeRO-3 add parameter all-gathers on top and raise the traffic further.

Mixture-of-experts all-to-all

With expert parallelism, each token's hidden state travels to the GPUs that hold its selected experts and back: two all-to-all exchanges per MoE layer. All-to-all loads every pair of GPUs at once, which is precisely what a switched NVLink fabric handles well and a PCIe tree handles poorly.

When PCIe is enough

• Data-parallel inference of a model that fits on one GPU. Each GPU serves its own copy of the model and a load balancer spreads the requests. The GPUs never exchange tensors, so throughput grows with the number of replicas, whatever connects them. Our guide to the best GPU for LLM inference shows which models fit on one card.
• Single-GPU work. LoRA fine-tuning of a model that fits, image and video generation, rendering, embeddings and most development work.
• Pipeline parallelism. When a model does need several GPUs, splitting it by layers sends only activations across each boundary: for the same 8K-token prompt on a 70B-class model, about 134 MB per boundary, roughly 2 ms at PCIe Gen5 speeds, against 37.6 GB per GPU with tensor parallelism. The price is latency, since the stages work in sequence for each request.
• Embarrassingly parallel jobs. Batch inference, hyperparameter sweeps and render farms split cleanly across GPUs.

In HGX and PCIe servers alike, GPUs talk to the host over PCIe to load data and weights, so host-side PCIe matters everywhere. What NVLink removes from the bus is GPU-to-GPU traffic.

NVLink and PCIe servers in our lineup

We rent both kinds of 8-GPU servers, and the interconnect is a large part of the difference between them.

The RTX PRO 6000 server is the lowest-priced 8-GPU server in our catalog, with 768 GB of GPU memory, and for the right workloads the missing NVLink costs nothing. It suits:

• Eight independent replicas of models that fit in 96 GB, such as a 70B model in FP8 on each card.
• Rendering, image and video generation, and other per-GPU work.
• Pipeline-parallel serving of models too large for one GPU.

Choose an HGX server instead when you rely on tensor parallelism at scale, train models that must be sharded across GPUs, run expert-parallel MoE models, or plan multi-node training: the HGX servers carry eight 400G or 800G adapters, against two 100 GbE ports on the PCIe server. Our RTX PRO 6000 vs H100 comparison covers the single-GPU side of the same choice.

Beyond one server: InfiniBand and RoCE

Outside NVL72 racks, NVLink stops at the edge of the server. Between servers, GPUs communicate through network adapters using RDMA, over InfiniBand or over Ethernet with RoCE (RDMA over Converged Ethernet). A 400G adapter carries 50 GB/s per direction, so even with one adapter per GPU, each GPU gets about a ninth of its NVLink 4 bandwidth when it talks to another server.

That gap dictates the standard layout for multi-node training: keep tensor parallelism inside a server, where NVLink is, and use pipeline and data parallelism across servers. NCCL (RCCL on AMD) picks the transport automatically, NVLink inside the node and the network between nodes; setting NCCL_DEBUG=INFO shows what it chose, and NCCL_IB_HCA and NCCL_SOCKET_IFNAME steer it when a server has several interfaces. Private networking between your servers is available on request, and our multi-GPU and NCCL guide walks through the configuration.

How to check what connects your GPUs

Do not take a spec sheet's word for it. Three commands tell you what you actually have on an NVIDIA server:

# Connection matrix: NV# means NVLink; PIX, PXB, PHB, NODE and SYS are
# paths of increasing distance through PCIe switches, host bridges and CPUs
nvidia-smi topo -m

# State and speed of each NVLink link (HGX servers)
nvidia-smi nvlink --status

# Measured all-reduce bandwidth across all eight GPUs, from NVIDIA's nccl-tests
./build/all_reduce_perf -b 8M -e 4G -f 2 -g 8

In the all-reduce output, the "busbw" column is normalized to be comparable with link bandwidth, so it shows directly whether you are getting NVLink-class or PCIe-class performance. On the MI355X server, rccl-tests provides the same benchmark. Run it once at delivery and again whenever performance looks off: a degraded link is far easier to spot here than in a training log.

Frequently asked questions

Is NVLink faster than PCIe?

Yes, by a wide margin. NVIDIA lists 900 GB/s per GPU for NVLink 4 on the H100 and H200 and 1.8 TB/s for NVLink 5 on the B200 and B300, against about 128 GB/s in both directions for a PCIe Gen5 x16 slot.

Do I need NVLink for LLM inference?

Only if the model must be split across GPUs with tensor parallelism. If the model fits on one GPU, running one copy per GPU needs no GPU-to-GPU traffic, and PCIe is fine.

Does the RTX PRO 6000 support NVLink?

No. The RTX PRO 6000 Blackwell Server Edition connects over PCIe Gen5 and has no NVLink, so our 8× RTX PRO 6000 server suits independent replicas, rendering and pipeline-parallel serving better than heavy tensor parallelism.

What is NVSwitch?

NVSwitch is the switch chip on NVIDIA HGX baseboards that connects all eight GPUs to each other at full NVLink bandwidth, so every pair can communicate at the same time. NVL72 racks extend the same idea to a 72-GPU NVLink domain.

Can I train across several servers without NVLink between them?

Yes. NVLink connects GPUs inside a server or an NVL72 rack, and traffic between servers goes over InfiniBand or RoCE network adapters. Keep tensor parallelism inside a server and use data or pipeline parallelism across servers.