nvidia-smi monitoring commands

These nvidia-smi monitoring commands cover what you need to monitor GPU usage on Linux, from a live view during a training run to metrics you keep for weeks. They work the same on single-GPU plans and 8-GPU servers. The last sections cover hardware errors, when to contact us, and the AMD equivalents.

Watch GPUs live

nvidia-smi              # one snapshot
nvidia-smi -l 2         # print a new snapshot every 2 seconds
watch -n 2 nvidia-smi   # same, redrawn in place

GPU-Util is the share of time at least one kernel was running, not how busy every core is, so 100% does not prove the GPU is used efficiently. Memory-Usage is allocated memory; frameworks such as vLLM reserve most of it on purpose. Pwr:Usage/Cap and Temp show power draw and temperature, and Perf the performance state, P0 being the highest. The process list shows which PIDs use each GPU; inside containers it is often empty, so check from the host. If nvidia-smi itself fails, see driver troubleshooting.

Stream metrics with dmon

nvidia-smi dmon -s pucm -d 5
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

dmon prints one line per GPU every 5 seconds (-d 5). The -s letters choose the metric groups: p power and temperature, u utilization, c clocks and m memory; add e for ECC and PCIe errors, t for PCIe throughput. -i 0,1 limits it to some GPUs. The second command lists each process with the GPU memory it holds.

Log to CSV with --query-gpu

nvidia-smi --query-gpu=timestamp,index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw,clocks.sm \
  --format=csv -l 10 >> ~/gpu-metrics.csv

This appends one line per GPU every 10 seconds; run it in tmux or as a service. Use --format=csv,noheader,nounits for files you parse with scripts. nvidia-smi --help-query-gpu lists every field, such as temperature.memory on HBM GPUs.

nvtop, an htop for GPUs

sudo apt install -y nvtop
nvtop

nvtop draws utilization and memory charts for each GPU above a process list. Recent versions, such as the one in Ubuntu 24.04, also support AMD GPUs. Press Q to quit.

Long-term metrics with DCGM exporter and Prometheus

For history and alerts, NVIDIA's DCGM exporter publishes GPU metrics in Prometheus format, and Grafana can chart them. A minimal start with Docker, using the current tag from NVIDIA's NGC catalog in place of TAG:

docker run -d --name dcgm-exporter --gpus all --cap-add SYS_ADMIN \
  -p 127.0.0.1:9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:TAG
curl -s http://127.0.0.1:9400/metrics | grep -E 'DCGM_FI_DEV_(GPU_UTIL|GPU_TEMP|POWER_USAGE|XID_ERRORS)'

You should see one value per GPU for each metric. Scrape it from your Prometheus server over private networking or an SSH tunnel, and alert on DCGM_FI_DEV_XID_ERRORS and on temperature. Never publish the metrics port to the internet. On AMD servers, AMD's device metrics exporter for Instinct GPUs plays the same role.

Temperature, power and ECC errors

nvidia-smi -q -d TEMPERATURE
nvidia-smi -q -d POWER,PERFORMANCE
nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER

• Temperature. Compare the current reading with the "GPU Slowdown Temp" and "GPU Shutdown Temp" thresholds in the same output. Readings near the slowdown threshold reduce clocks.
• Power and clocks. The PERFORMANCE section lists clock event reasons. "SW Power Cap" under heavy load is normal; "HW Slowdown" or "HW Thermal Slowdown" while the GPU is busy is worth reporting.
• ECC. Data center GPUs correct single-bit memory errors on the fly. Volatile counters reset when the driver reloads; aggregate counters cover the GPU's lifetime. Uncorrectable errors usually stop the running job.
• Row remapping. HBM GPUs replace faulty memory rows. "Pending: Yes" means the GPU needs a reset or reboot to apply the remap. Repeated remaps or a remapping failure mean the GPU needs attention.

Xid errors and when to contact us

The NVIDIA driver reports GPU faults in the kernel log as Xid messages:

sudo dmesg -T | grep -i xid
sudo journalctl -k --since "24 hours ago" | grep -i xid

When you contact us, include the server IP, the Xid lines and an NVIDIA bug report, which gathers the relevant logs in one file:

sudo nvidia-bug-report.sh

It writes nvidia-bug-report.log.gz to the current directory. Failed GPUs are replaced as a priority under our service level agreement, and engineers answer 24/7 through the contact form.

AMD equivalents with rocm-smi and amd-smi

More on the AMD tools in ROCm setup.