GPU and AI infrastructure glossary

Trillions of operations per second, a peak throughput figure that vendors quote for AI workloads. It is usually measured at low precision, such as INT8 or FP4, and often assumes sparsity, so AI TOPS figures are only comparable when those assumptions match. NVIDIA, for example, lists the GeForce RTX 5090 at 3,352 AI TOPS, measured at FP4 with sparsity.

A collective communication operation in which every GPU contributes a buffer, the buffers are combined, usually by summing, and every GPU receives the result. In data-parallel training, an all-reduce averages gradients across GPUs after each step, so its speed depends heavily on the interconnect: NVLink inside a server, InfiniBand or RoCE between servers. Libraries such as NCCL and RCCL implement it.

A physical server rented without a hypervisor between you and the hardware. Your operating system runs directly on the machine, with access to every GPU, CPU core, disk and network card and no virtualization overhead. Bare metal suits multi-GPU training and any workload that needs low-level control of drivers and kernel settings. FullGPU's 8-GPU servers are bare-metal machines.

Bfloat16, or brain floating point: a 16-bit number format with 8 exponent bits and 7 mantissa bits. It keeps the numeric range of FP32 with less precision, which keeps training stable without the loss scaling that FP16 often needs. BF16 is the default precision for most large-model training and serving, and weights stored in BF16 take about 2 bytes per parameter.

NVIDIA's GPU architecture after Hopper, used in the B200, the RTX PRO 6000 Blackwell and the GeForce RTX 50 series. Data-center Blackwell GPUs combine two dies into one GPU and introduce fifth-generation NVLink and hardware support for FP4. Blackwell Ultra, used in the B300 and the GB300 NVL72, raises memory to 288 GB of HBM3e per GPU. See our B200 vs B300 comparison.

NVIDIA's parallel computing platform and programming model, including the compiler, runtime and libraries such as cuBLAS and cuDNN. PyTorch and most other AI frameworks run on NVIDIA GPUs through CUDA. A framework built for a given CUDA version needs an NVIDIA driver recent enough to support it, so driver and CUDA versions have to be chosen together. AMD's counterpart is ROCm. See our NVIDIA drivers and CUDA guide.

NVIDIA's name for the general-purpose arithmetic units inside each streaming multiprocessor of its GPUs. They execute standard FP32 and integer instructions, while the matrix math that dominates AI workloads runs mostly on Tensor Cores. CUDA core counts help when comparing GPUs of the same architecture, but they say little about AI performance across generations.

A training strategy in which each GPU holds a full copy of the model and processes a different slice of every batch. After the backward pass, gradients are averaged across GPUs with an all-reduce so that all copies stay identical. It is the simplest way to scale LLM training, but each GPU must hold the full model, its gradients and optimizer state unless they are sharded with FSDP or ZeRO.

NVIDIA Data Center GPU Manager, a suite of tools for monitoring and managing data-center GPUs. It reports health, temperatures, power, clocks, memory errors and NVLink status, runs diagnostics that test a GPU before or after a long job, and can export metrics to monitoring systems such as Prometheus. Operators use it to catch failing hardware early. See our GPU monitoring guide.

Protection against distributed denial-of-service attacks: floods of traffic from many sources that aim to saturate a server or its network link. Mitigation systems detect abnormal traffic and filter it before it reaches the server, while legitimate traffic continues to flow. It complements a host firewall rather than replacing one. FullGPU applies DDoS mitigation to every public IP address.

Outbound data traffic leaving a server or network, such as responses to users, model downloads or datasets copied elsewhere. Many cloud providers bill egress per gigabyte, which adds up when you serve models or move large checkpoints. FullGPU includes 20 TB per month on single-GPU plans and 50 TB on 8-GPU servers with no egress fees, under fair use. See bandwidth and egress.

A way to distribute a Mixture of Experts model by placing different experts on different GPUs. Each token is sent to the GPUs that hold its selected experts and the results are sent back, which creates all-to-all traffic between GPUs at every MoE layer. Expert parallelism lets very large MoE models run across a server or a cluster, and it benefits from a fast interconnect such as NVLink.

A 4-bit floating-point format used for model weights and, on supporting hardware, for matrix math. Schemes such as NVFP4 and MXFP4 attach a shared scale factor to each small block of values to preserve accuracy. FP4 weights take about half a byte per parameter, so a 70B model needs roughly 35 GB before KV cache and overhead. Blackwell GPUs and AMD's MI355X support FP4 in hardware, while Hopper GPUs do not; see H200 vs B200.

An 8-bit floating-point format with two common variants: E4M3, with more precision, and E5M2, with more range. According to NVIDIA's published specifications, Hopper and Blackwell GPUs run FP8 matrix math at twice their peak BF16 rate, and FP8 weights take about 1 byte per parameter. FP8 is widely used for inference and increasingly for training. Read FP8 and FP4 precision explained.

Double-precision floating point, with 64 bits per value. Codes for computational fluid dynamics, climate modeling and quantum chemistry often depend on it, while AI training and inference almost never do. FP64 throughput varies widely between data-center GPUs, and consumer and workstation cards deliver only a small fraction of their FP32 rate, so check the vendor's FP64 figure before choosing hardware for research and HPC.

Fully Sharded Data Parallel, PyTorch's implementation of sharded data parallelism. Instead of keeping a full copy of the model on every GPU, FSDP splits parameters, gradients and optimizer state across GPUs and gathers each layer's parameters only while that layer is computed. This makes it possible to fully fine-tune a 70B model across several GPUs, at the cost of extra communication between them.

The seventh generation of graphics double data rate memory, used on GPUs such as the GeForce RTX 5090 and the RTX PRO 6000 Blackwell. GDDR7 chips sit on the circuit board around the GPU rather than inside its package, which keeps cost down but limits bandwidth: NVIDIA lists about 1.8 TB/s for the RTX 5090, against 8 TB/s of HBM3e bandwidth on the B200.

A virtualization technique that assigns a whole physical GPU to one virtual machine, typically through PCIe passthrough with an IOMMU. The guest operating system loads the GPU's own driver and gets close to bare-metal performance, with all of the GPU's memory and compute. FullGPU's single-GPU servers use passthrough: a dedicated VM with the entire GPU assigned to it.

One GPU used for one hour, the common unit for pricing GPU time and estimating jobs: 8 GPUs running for 10 hours use 80 GPU-hours. To compare a monthly price with an hourly one, divide the monthly price by the number of GPUs and by 730, the average number of hours in a month. A single H100 at $2,190 a month works out to about $3.00 per GPU-hour. See monthly vs hourly GPU pricing.

High Bandwidth Memory 3E, an enhanced generation of HBM3. HBM stacks DRAM dies vertically and places the stacks in the same package as the GPU, linked by very wide interfaces, which delivers far more bandwidth than GDDR memory. Per NVIDIA, HBM3e gives the H200 141 GB at 4.8 TB/s and the B300 288 GB at about 8 TB/s; AMD's MI355X also carries 288 GB of HBM3E.

The generation of High Bandwidth Memory after HBM3e. HBM4 doubles the interface width of each stack to 2,048 bits, which raises bandwidth per stack. It is the memory of NVIDIA's Rubin GPUs: the Vera Rubin NVL72 combines 72 Rubin GPUs, each with 288 GB of HBM4, with 36 Vera CPUs. Systems built on HBM4 are expected in limited availability in late 2026.

NVIDIA's server platform for data-center GPUs: a baseboard carrying four or eight SXM GPUs connected by NVLink, with NVSwitch chips on 8-GPU boards, around which manufacturers build complete servers. In an 8-GPU HGX server such as the HGX B200, every GPU can communicate with every other at full NVLink bandwidth, which large training and inference jobs rely on.

NVIDIA's data-center GPU architecture before Blackwell, used in the H100 and H200. Hopper introduced the Transformer Engine with FP8 Tensor Core support, fourth-generation NVLink at 900 GB/s per GPU, and HBM3 memory, followed by HBM3e on the H200. Its software stack is mature, and it remains widely used for training and inference. See our H100 vs H200 comparison.

A high-bandwidth, low-latency network standard widely used to connect GPU servers in training clusters. It supports RDMA natively, letting network cards move data directly between the memory of different servers, including GPU memory, without involving the CPU in each transfer. Current generations run at 400 or 800 Gb/s per port, and multi-node GPU clusters often use it for their back-end network.

AMD's interconnect technology, used inside its processors and between GPUs. On an 8-GPU AMD Instinct MI355X platform, each GPU connects directly to the other seven over Infinity Fabric links, which AMD lists at 153.6 GB/s each. It fills the role NVLink plays on NVIDIA systems, using direct all-to-all links instead of switch chips.

Memory that stores the attention keys and values of tokens a language model has already processed, so they are not recomputed for every new token. It grows with context length, the number of concurrent requests and the model's layer and head dimensions, and it often limits how many users one GPU can serve. Inference engines such as vLLM manage it in blocks to reduce waste. See how much VRAM LLMs need.

Low-Rank Adaptation, a parameter-efficient fine-tuning method. LoRA freezes the original weights and trains small low-rank matrices added to selected layers, often less than 1% of the model's parameters. Because gradients and optimizer state are kept only for these adapters, it needs far less GPU memory than full fine-tuning, and the adapters are small files that can be swapped at inference time. See GPUs for fine-tuning.

The rate at which a GPU reads and writes its own memory, measured in gigabytes or terabytes per second. When a language model generates text at small batch sizes, each new token requires reading the model's weights from memory, so bandwidth often limits speed more than compute does. Per vendor specifications, it ranges from 864 GB/s on the L40S to about 8 TB/s on the B200, B300 and MI355X. See the best GPU for LLM inference.

Multi-Instance GPU, an NVIDIA feature that splits a supported data-center GPU into as many as seven isolated instances, each with its own share of memory, cache and compute. It lets one GPU serve several small workloads with hardware isolation. FullGPU does not partition GPUs: each GPU you rent is whole, so all of its memory and compute are yours. See dedicated vs shared GPUs.

Training or inference that combines number formats: most matrix math runs in a low precision such as BF16 or FP8, while values that are sensitive to rounding, such as master weights, optimizer state and some accumulations, stay in FP32. Mixed precision reduces memory use and speeds up Tensor Core math with little or no loss of accuracy, and frameworks such as PyTorch automate most of the bookkeeping.

A model architecture in which each MoE layer contains many expert sub-networks and a router sends each token to only a few of them. Compute per token stays modest while total parameters grow: according to DeepSeek, DeepSeek-V3 has 671 billion parameters but activates about 37 billion per token. All experts must still fit in GPU memory, which drives the hardware requirements of these models.

The NVIDIA Collective Communications Library, pronounced "nickel". It implements all-reduce, all-gather, broadcast and other collective operations across GPUs, choosing the fastest available path: NVLink inside a server, InfiniBand or RoCE between servers. PyTorch uses it for multi-GPU training on NVIDIA hardware, and AMD's counterpart is RCCL. See our multi-GPU and NCCL guide.

NVIDIA's rack-scale design that joins 72 GPUs into a single NVLink domain, with NVSwitch trays linking every GPU to every other inside one liquid-cooled rack. The GB300 NVL72 pairs 72 Blackwell Ultra GPUs with 36 Grace CPUs and about 130 TB/s of total NVLink bandwidth, per NVIDIA. The Vera Rubin NVL72 applies the same design to Rubin GPUs and Vera CPUs.

NVIDIA's high-speed interconnect between GPUs. It moves data far faster than PCIe: per NVIDIA, fourth-generation NVLink on Hopper provides 900 GB/s per GPU and fifth-generation NVLink on Blackwell 1.8 TB/s. It matters most for tensor parallelism and other workloads that exchange data between GPUs at every layer. PCIe cards such as the RTX PRO 6000 Blackwell and the RTX 5090 do not have it. See NVLink vs PCIe.

A switch chip that connects many GPUs over NVLink so that each one can talk to every other at full bandwidth at the same time. In 8-GPU HGX servers, NVSwitch chips on the baseboard form the GPU fabric; in NVL72 racks, dedicated switch trays extend it to 72 GPUs. Without switches, each GPU needs direct links to the others, which limits how many GPUs can be fully connected.

OCP Accelerator Module, an open form factor for accelerator modules defined by the Open Compute Project. AMD Instinct GPUs such as the MI300X and MI355X use it, with eight modules mounted on a universal baseboard. OAM is the open counterpart of NVIDIA's SXM: both put the GPU on a module that plugs into a board, which allows higher power and direct GPU-to-GPU links. See MI355X vs B200.

The fifth generation of PCI Express, the standard bus between CPUs, GPUs, network cards and NVMe drives. An x16 Gen5 link carries about 64 GB/s in each direction, twice Gen4. PCIe GPUs such as the RTX PRO 6000 Blackwell and the RTX 5090 use it to reach the CPU and, without NVLink, to exchange data with other GPUs, so multi-GPU communication is slower than on SXM systems. See RTX PRO 6000 vs H100.

A model-parallel strategy that splits a model's layers into sequential stages on different GPUs or servers. Micro-batches flow through the stages like an assembly line, so several stages work at once. It needs less bandwidth than tensor parallelism, which makes it practical across servers, but idle time while the pipeline fills and drains, called the pipeline bubble, reduces efficiency.

Quantized LoRA: fine-tuning with LoRA adapters while the frozen base model is stored in 4-bit precision, typically in the NF4 format. Only the small adapters are trained in higher precision. QLoRA cuts memory enough that its authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU, at some cost in speed compared with LoRA on BF16 weights.

Storing a model's weights, and sometimes its activations and KV cache, in fewer bits, such as FP8, INT8, INT4 or FP4 instead of BF16. Memory falls roughly in proportion to the bits removed, and throughput can rise on hardware that supports the lower precision. The trade-off is some loss of accuracy, which methods such as AWQ, GPTQ or FP8 calibration keep small. See how low-precision formats compare.

Remote Direct Memory Access: a way for one machine to read or write another machine's memory over the network without involving the remote CPU or the operating system's network stack. With GPUDirect RDMA, network cards move data straight to and from GPU memory, which cuts latency and CPU load in multi-node training. InfiniBand and RoCE both provide RDMA.

RDMA over Converged Ethernet, which carries RDMA traffic over Ethernet networks. RoCE v2 runs over routable UDP/IP and, on a well-tuned lossless or congestion-controlled fabric, can deliver performance comparable to InfiniBand using Ethernet switches. It is common in AI clusters built on Ethernet, and FullGPU's AMD Instinct MI355X servers have eight 400G RoCE network interfaces.

AMD's open-source software stack for GPU computing and the counterpart of CUDA. It includes drivers, the HIP programming interface, math libraries and RCCL for collective communication, and it is supported by PyTorch, vLLM and other major frameworks. FullGPU provides ROCm images for its AMD Instinct servers; see our AMD ROCm guide.

NVIDIA's module form factor for data-center GPUs. Instead of a PCIe card, the GPU sits on a module mounted directly on the server's baseboard, which allows higher power limits, such as 700 W for the H100 SXM, and NVLink connections to the other GPUs. HGX servers use SXM GPUs, while PCIe versions of the same chips usually run at lower power and clocks.

Thermal design power: the maximum sustained power, in watts, that a chip is designed to draw and its cooling to remove. Vendors list about 700 W for an H100 SXM, 575 W for a GeForce RTX 5090 and 1,400 W for an AMD Instinct MI355X, which is liquid-cooled. Power ratings drive the electrical and cooling capacity a data center needs, which is why the newest high-power GPUs increasingly rely on liquid cooling.

Specialized units in NVIDIA GPUs that perform matrix multiply-accumulate operations, the core math of neural networks, far faster than general-purpose cores. Each generation has added lower-precision formats: FP8 arrived with Hopper and Ada Lovelace, and FP4 with Blackwell. AMD's equivalent units are called Matrix Cores. Vendor AI throughput figures in TFLOPS or PFLOPS refer to these units. See H100 vs B200.

A model-parallel strategy that splits the weight matrices of each layer across several GPUs, so every GPU computes part of every layer. Partial results are combined with collective operations at each layer, which demands high bandwidth and low latency, so tensor parallelism is usually kept within one NVLink-connected server. Inference engines use it to serve models too large for one GPU; see our guide to self-hosting LLM inference with vLLM.

Trillions (TFLOPS) and quadrillions (PFLOPS) of floating-point operations per second; 1 PFLOPS equals 1,000 TFLOPS. Vendors quote peak figures for each precision, often as a dense and a sparse number, where the sparse figure assumes 2:4 structured sparsity and is usually twice the dense one. NVIDIA lists the H100 SXM at 989 TFLOPS of dense BF16. Real workloads reach only part of the peak. Compare models on our GPU overview.

A way of sharing one GPU among several processes or tenants by giving each a turn in rapid succession. Unlike MIG, time-slicing provides no memory or fault isolation between workloads, and each workload's performance depends on what else is running. FullGPU does not time-share GPUs: each GPU you rent runs only your workloads.

The rate at which a language model processes or generates tokens, the units of text it reads and writes. It is reported either per user, which measures how fast one response streams, or as total throughput across all concurrent requests on a GPU. The two pull in opposite directions: larger batches raise total throughput but can slow each stream, so check which one a figure measures. See GPUs for inference.

Virtual GPU technology that divides one physical GPU into several virtual GPUs assigned to different virtual machines, each with a fixed share of memory and a time-shared or partitioned share of compute. It suits virtual desktops and light workloads. FullGPU does not use vGPU: single-GPU plans pass the entire physical GPU through to a dedicated VM, as explained on our trust and security page.

An open-source inference and serving engine for large language models. It manages the KV cache in fixed-size blocks with PagedAttention, batches incoming requests continuously, and supports tensor parallelism, quantized models and an OpenAI-compatible server. vLLM runs on NVIDIA GPUs with CUDA and on AMD GPUs with ROCm. See our PyTorch and vLLM guide.

The memory attached to a GPU: HBM on most data-center GPUs, and GDDR on workstation, consumer and some data-center cards such as the L40S. Model weights, KV cache and activations must all fit in it. As a rough rule, weights need parameters × bytes per parameter, so a 70B model takes about 140 GB in BF16, plus 10–30% headroom for KV cache and overhead. Try our VRAM calculator.

Zero Redundancy Optimizer, a family of memory optimizations in Microsoft's DeepSpeed library. Stage 1 shards optimizer state across data-parallel GPUs, stage 2 also shards gradients, and stage 3 shards the parameters themselves, so memory per GPU falls as GPUs are added. ZeRO-Offload and ZeRO-Infinity can also move state to CPU memory or NVMe. PyTorch FSDP applies the same idea.