FP8 vs FP4 vs BF16: number formats for AI, explained

FP8 vs FP4 vs BF16 is a trade between memory, speed and accuracy. BF16 uses two bytes per value and remains the default for training, and for inference when quality comes first. FP8 uses one byte and is widely used for inference on recent GPUs. FP4 uses half a byte, only works with block scaling, and runs natively only on the newest hardware. This guide explains the bit layouts, the block-scaled formats (NVFP4, MXFP4 and MXFP6), which GPUs support what, and how to choose.

FP8 vs FP4 vs BF16 at a glance

A floating-point number has a sign bit, exponent bits and mantissa bits. The table lists the formats you will meet in AI work, from widest to narrowest.

Range versus precision: exponent and mantissa bits

Exponent bits set the range: how large and how small a value can get. Mantissa bits set the precision: how finely values are spaced within that range. Every format is a different split of the same budget.

• BF16 keeps FP32's 8 exponent bits, so it covers the same range, and cuts the mantissa to 7 bits, about two to three significant decimal digits. Because the range matches FP32, training needs no loss scaling, which is why BF16 has largely replaced FP16 for training.
• FP16 has more precision (10 mantissa bits) but tops out at 65,504, and small gradients underflow to zero, so FP16 training needs loss scaling.
• TF32 is not a storage format. It is a tensor-core mode, introduced with NVIDIA's Ampere generation, that takes FP32 inputs, multiplies them with a 10-bit mantissa and accumulates in FP32. In PyTorch, torch.set_float32_matmul_precision("high") allows it.

Matrix multiplications on tensor cores take low-precision inputs but accumulate in higher precision, typically FP32. The low-precision part is what you store and move; the accumulation protects the sum.

FP8: E4M3 and E5M2

FP8 comes in two variants. E4M3 has more precision and a maximum of 448, so it is used for weights and activations. E5M2 trades a mantissa bit for range, up to 57,344, which suits gradients in FP8 training. NVIDIA's Transformer Engine, for example, uses E4M3 in the forward pass and E5M2 for gradients in its hybrid recipe.

Because the range is so narrow, FP8 values almost always travel with a scale factor: a tensor is rescaled into the representable range before it is cast to FP8, and the scale is stored so the values can be restored. The scale can be per tensor, per channel, per token or per block. Granularity matters: with one scale per tensor, a single outlier forces a large scale and pushes small values to zero. DeepSeek-V3, for instance, was released with FP8 weights carrying one scale per 128 × 128 block; our article on DeepSeek R1 hardware requirements shows what that means for memory.

FP6, FP4 and block scaling: NVFP4 vs MXFP4

FP4 in the E2M1 layout can represent only 0, 0.5, 1, 1.5, 2, 3, 4 and 6, plus their negatives. With so few levels, one scale per tensor is hopeless. Block-scaled formats give every small block of values its own scale, stored next to the data.

The difference between the two FP4 formats is easiest to see on a single block. Say its largest magnitude is 0.42. NVFP4 can choose a scale close to 0.42 / 6 = 0.07, so 0.42 lands almost exactly on the top code, 6. MXFP4 must use a power of two: the reference conversion in the OCP specification picks 2⁻⁴ = 0.0625, which maps 0.42 to 6.72, above the largest code, so it is clipped to 6 and decodes as 0.375. Smaller blocks and finer scales let NVFP4 fit each block more closely, at the cost of a quarter bit more per value. Power-of-two scales are cheaper to apply in hardware and cover a much wider range.

Which GPUs support which formats

"Supported" here means the tensor cores compute in that format natively. Any GPU can store 4-bit weights and expand them to 16-bit before multiplying, the approach behind weight-only methods such as GPTQ and AWQ, but that saves memory and bandwidth, not compute.

NVIDIA's data-center Blackwell GPUs, the B200 and B300, support both NVFP4 and the OCP MX formats. In practice, FP8 is the lowest floating-point precision that Hopper and Ada tensor cores run natively, while Blackwell and the MI355X add FP6 and FP4. For a concrete case, see our H200 vs B200 comparison. Every GPU in our single-GPU and 8-GPU lineup supports FP8 in hardware.

Memory savings, worked out

For a 70B-parameter model, weights alone:

Real checkpoints come out slightly larger, because embeddings, the output layer and normalization weights often stay in 16-bit. Two other things do not shrink with the weights: activations, and the KV cache unless you quantize it too. In memory-bound decode, fewer bytes per weight also means fewer bytes to stream per token, so lower precision speeds up generation as well as saving memory. For the full sizing method, see how much VRAM an LLM needs.

Accuracy: weights, activations and the KV cache

You can quantize three different things, and they behave differently.

• Weights are fixed after training, so they can be quantized once, offline, with care. Weight-only 4-bit quantization with 16-bit activations (W4A16) is a long-established approach.
• Activations change with every input and contain outliers: a few channels with magnitudes far larger than the rest. They need dynamic scaling or calibration. Quantizing both weights and activations to FP8 (W8A8) lets the matrix multiplications run on FP8 tensor cores; W4A4 with NVFP4 or MXFP4 is the most aggressive common setup.
• The KV cache can be stored in FP8 to halve its memory, which buys concurrency or context length. Check long-context quality when you do.

Calibration means running a small set of representative inputs through the model to choose the scales. Toolkits such as NVIDIA's TensorRT Model Optimizer, llm-compressor and AMD Quark automate it. Evaluate the result on the tasks you care about, including long contexts, math and code, rather than on perplexity alone, and always against a BF16 baseline. The lower the bit width, the more the outcome depends on the specific model and task.

Training is more conservative than inference. BF16 mixed precision is the standard; FP8 training is established on Hopper and Blackwell through libraries such as Transformer Engine; 4-bit training is still an active research area.

Why vendors quote dense and sparse figures

Spec sheets usually give two compute numbers per format. Vendor-published peak figures per GPU:

The sparse figures assume structured sparsity. On NVIDIA GPUs that is the 2:4 pattern: in every group of four consecutive weights, two are zero, so the tensor cores can skip half the multiplications. Most LLMs are served dense, so plan with the dense column. Compare like with like, since one headline may be sparse and another dense, and check the ratio rather than assuming it: the B300's FP4 figures (13.5 dense, 18 sparse) are not a clean 2×. Peak numbers are ceilings that real workloads reach only in part. Note also the pattern across each row: per these vendor figures, each halving of precision at least doubles dense peak throughput, which is the other half of the case for FP8 and FP4.

Practical advice: which precision to use

• Pre-training and full fine-tuning: BF16 mixed precision, with FP32 master weights and optimizer state. Move to FP8 for large runs on H100 or newer once the loss curve matches a BF16 run.
• Fine-tuning on a budget: QLoRA, which keeps the base model in 4-bit NF4 and trains 16-bit adapters.
• Serving on Hopper or Ada (H100, H200, L40S): FP8 weights and activations, plus an FP8 KV cache when you need more concurrency.
• Serving on Blackwell (B200, B300, RTX PRO 6000, RTX 5090): start with FP8, then try NVFP4 where memory or bandwidth is the bottleneck and your evaluations hold.
• Serving on the MI355X: FP8 first, then MXFP4 or MXFP6 with the same validation.

In code, the first steps are small:

# PyTorch: allow TF32 tensor cores for FP32 matrix multiplications
torch.set_float32_matmul_precision("high")

# vLLM: quantize weights to FP8 at load time and keep the KV cache in FP8
vllm serve <model> --quantization fp8 --kv-cache-dtype fp8

Whatever you pick, keep a BF16 baseline and an evaluation set, and compare before you ship. The VRAM calculator shows how each format changes the GPU you need, and our guide to the best GPU for LLM inference maps model sizes to hardware.

Frequently asked questions

Is FP8 better than BF16?

FP8 uses half the memory and, on GPUs with FP8 tensor cores, offers twice the peak throughput of BF16 per vendor figures. BF16 is more precise, so it remains the default for training, and FP8 inference should be validated against a BF16 baseline.

Does the H100 support FP4?

No. Hopper GPUs such as the H100 and H200 support FP8 but have no FP4 or FP6 tensor cores; those arrived with Blackwell. You can still store 4-bit weights on an H100 with weight-only quantization, which saves memory but computes in 16-bit.

What is the difference between NVFP4 and MXFP4?

Both store values as 4-bit E2M1 numbers with a shared scale per block. NVFP4 uses blocks of 16 values with an FP8 E4M3 scale, while MXFP4 uses blocks of 32 values with a power-of-two scale, so NVFP4 fits each block more closely at a slightly higher storage cost.

How much memory does FP4 save compared with BF16?

Close to four times for the weights: a 70B model drops from about 140 GB in BF16 to roughly 37 to 39 GB in MXFP4 or NVFP4, block scales included. The KV cache and activations are separate and shrink only if you quantize them too.

Should I train in FP8?

For large training runs on H100, H200 or Blackwell GPUs, FP8 can cut memory traffic and raise throughput, and libraries such as NVIDIA Transformer Engine handle the scaling. Validate the loss curve against a BF16 run first, since BF16 remains the safe default.