Check NVIDIA driver and CUDA version

To check NVIDIA driver and CUDA version on a GPU server you need two commands, because they answer different questions: nvidia-smi reports the driver, and nvcc reports the CUDA toolkit you compile with. This page explains both, how they must line up, how to install another CUDA toolkit on Ubuntu 24.04 or 22.04 side by side, and how to fix the errors you are most likely to meet.

Read the driver and CUDA version in nvidia-smi

nvidia-smi
nvidia-smi --query-gpu=index,name,driver_version,memory.total --format=csv
cat /proc/driver/nvidia/version

The header of nvidia-smi shows three versions:

• NVIDIA-SMI is the version of the tool itself, normally the same as the driver.
• Driver Version is the installed driver.
• CUDA Version is the newest CUDA release this driver supports. It does not mean that toolkit is installed.

/proc/driver/nvidia/version shows the kernel module that is actually loaded. If it differs from the installed driver packages, for example after an upgrade, a reboot is pending.

Check the CUDA toolkit with nvcc

nvcc --version
ls -d /usr/local/cuda*

The line "Cuda compilation tools, release X.Y" gives the toolkit version. /usr/local/cuda points to the default toolkit, and other versions live in /usr/local/cuda-X.Y. If nvcc is not found, add the toolkit to your environment (append the lines to ~/.bashrc to keep them):

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Frameworks often ignore this toolkit. PyTorch wheels and most containers ship their own CUDA libraries and only need a driver that is new enough, as the PyTorch and vLLM guide shows. Check what PyTorch sees with:

python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

How driver and toolkit versions fit together

• The driver must be at least as new as the code. The CUDA Version shown by nvidia-smi should be equal to or higher than the CUDA version your toolkit, wheel or container targets. Older toolkits run on newer drivers.
• Minor version compatibility has limits. Within one major release, such as 12.x, NVIDIA lets many applications built with a newer minor version run on an older driver of the same major release, but features such as PTX just-in-time compilation may fail. Test before you rely on it.
• The GPU architecture sets a floor. Hopper GPUs (H100, H200) work with CUDA 12 and later. Blackwell GPUs (B200, RTX PRO 6000, RTX 5090) need CUDA 12.8 or later and framework builds that include Blackwell; for B300, check NVIDIA's release notes for the minimum version.

Rule of thumb: keep the installed driver, and pick toolkits, wheels and container tags whose CUDA version is at or below the CUDA Version in nvidia-smi.

Install another CUDA toolkit on Ubuntu 24.04

You can add a toolkit side by side from NVIDIA's apt repository without replacing the driver. If /etc/apt/sources.list.d/ already has a CUDA repository entry, skip step 1.

1. Add NVIDIA's repository key. On 22.04, replace ubuntu2404 with ubuntu2204:

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   sudo apt update

2. List the toolkit versions available:

   apt-cache search --names-only '^cuda-toolkit-[0-9]+-[0-9]+$'

3. Install the toolkit-only package for the version you need, for example 12.8:

   sudo apt install -y cuda-toolkit-12-8

   Always use cuda-toolkit-X-Y. The cuda, cuda-X-Y and cuda-drivers packages also install a driver.
4. Select it in your shell and check:

   export PATH=/usr/local/cuda-12.8/bin:$PATH
   export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
   nvcc --version

   You should see release 12.8. When the toolkits come from NVIDIA's packages, sudo update-alternatives --config cuda changes the default that /usr/local/cuda points to.

With NVIDIA's repository enabled, a later apt upgrade may offer newer driver packages. Read the list before you confirm, or hold the driver as described in OS images. If only one project needs another CUDA version, a container is often simpler: see Docker GPU containers.

Enable persistence mode

Persistence mode keeps the driver initialized when no process is using the GPU, so jobs start without waiting for the driver to load. Check it, then turn it on:

nvidia-smi --query-gpu=index,persistence_mode --format=csv
sudo nvidia-smi -pm 1

You should see "Enabled" for every GPU. The setting does not survive a reboot. If it is disabled after a restart, apply it at boot from the root crontab: run sudo crontab -e and add the line @reboot /usr/bin/nvidia-smi -pm 1.

Troubleshooting common errors

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

The kernel module is not loaded, often after a kernel update for which it was not built. Collect:

uname -r
lsmod | grep nvidia
dkms status
sudo dmesg | grep -i -E 'nvidia|nvrm'

If dkms status lists the NVIDIA module for other kernels but not for the one shown by uname -r, install the matching headers, rebuild and reboot:

sudo apt install -y linux-headers-$(uname -r)
sudo dkms autoinstall
sudo reboot

If dkms is not installed, the driver uses prebuilt modules: send us the output above through the contact form.

Failed to initialize NVML: Driver/library version mismatch

The driver packages were upgraded while the old module was loaded. Run sudo reboot. Afterwards, nvidia-smi and /proc/driver/nvidia/version should show the same version. Holding the driver packages prevents this.

CUDA error 802, system not yet initialized

On 8-GPU HGX servers, NVIDIA Fabric Manager must be running, at exactly the driver's version, before CUDA can use the NVSwitch fabric. Check systemctl status nvidia-fabricmanager, restart it with sudo systemctl restart nvidia-fabricmanager, and contact us if it does not start.

CUDA driver version is insufficient for CUDA runtime version

The program, wheel or container was built for a newer CUDA than the driver supports. Use a build for an older CUDA release, or ask us about a driver upgrade. For hardware errors (Xid messages in dmesg), see GPU monitoring.