Mount NVMe drives on Ubuntu

This guide shows how to mount NVMe drives on Ubuntu: find them, format and mount a single drive, or combine them in an NVMe RAID 0 array with mdadm, then check drive health and set up backups. Our 8-GPU servers have eight 3.84 TB NVMe data drives, or four on the RTX PRO 6000 server. Depending on your order, they may be delivered unmounted. On single-GPU plans, run the same checks: your NVMe storage may already be the system disk.

Find your NVMe drives

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL
sudo apt install -y nvme-cli
sudo nvme list

The system disk is the one with partitions mounted on / (and usually /boot/efi): never format it. Unused data drives show no partitions, no FSTYPE and no mount point. lsblk counts in binary units, so a 3.84 TB drive appears as about 3.5T. If the drives already show an FSTYPE such as linux_raid_member or ext4, they are set up: check cat /proc/mdstat and findmnt before you change anything.

Format and mount a single drive

1. Create a filesystem. ext4 is Ubuntu's default, and -m 0 skips the 5% space reserve for root, which a data volume does not need. XFS (sudo mkfs.xfs -L data1 /dev/nvme1n1, from the xfsprogs package) is a common alternative for large files.

   sudo mkfs.ext4 -m 0 -L data1 /dev/nvme1n1

2. Mount it:

   sudo mkdir -p /data
   sudo mount /dev/nvme1n1 /data

3. Make it permanent. The UUID does not change between boots, and nofail lets the server boot even if the drive is missing:

   echo "UUID=$(sudo blkid -s UUID -o value /dev/nvme1n1) /data ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   sudo findmnt --verify
   sudo systemctl daemon-reload

   For XFS, write xfs instead of ext4 in that line.
4. Test the entry without rebooting:

   sudo umount /data
   sudo mount -a
   df -h /data

   You should see /data with a size of about 3.5T.

To use more drives separately, repeat with /data2, /data3 and so on. If you work as a non-root user, take ownership of the mount point with sudo chown $USER: /data.

Combine the data drives in RAID 0 with mdadm

Instead of mounting drives one by one, you can combine them. RAID 0 stripes data across all drives: one large volume, about 28 TiB from eight drives, with their combined throughput. The trade-off is serious. If any one drive fails, the whole array is lost, so use it for data you can download or regenerate, and keep copies of anything valuable elsewhere. To survive a drive failure, RAID 10 (--level=10) mirrors every block at half the capacity.

1. Install mdadm and create the array from the data drives only. This example uses eight; on the RTX PRO 6000 server, use --raid-devices=4 and four names:

   sudo apt install -y mdadm
   sudo mdadm --create /dev/md0 --level=0 --raid-devices=8 \
     /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 \
     /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1

   If mdadm warns that a drive appears to contain data, stop and check lsblk again before you answer.
2. Check the array: cat /proc/mdstat should show md0 : active raid0 with every drive, and sudo mdadm --detail /dev/md0 a clean state.
3. Save the array definition so it assembles as /dev/md0 at boot:

   sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
   sudo update-initramfs -u

4. Create the filesystem and mount it as for a single drive, using /dev/md0. mkfs reads the stripe layout from the array, so no tuning is needed:

   sudo mkfs.ext4 -m 0 -L data /dev/md0
   sudo mkdir -p /data
   echo "UUID=$(sudo blkid -s UUID -o value /dev/md0) /data ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   sudo findmnt --verify && sudo systemctl daemon-reload && sudo mount -a
   df -h /data

   You should see a size of about 28T for eight drives.

Check drive health

sudo apt install -y smartmontools
sudo smartctl -a /dev/nvme1n1
sudo nvme smart-log /dev/nvme1n1

In the smart-log output, critical_warning and media_errors should be 0, available_spare should stay above available_spare_threshold, and percentage_used is the manufacturer's wear estimate. smartctl shows the same data with an overall PASSED or FAILED verdict. Inside a single-GPU VM, these readings may not be available. If a drive reports errors, contact us: failed components are replaced as a priority.

Back up what matters

Local storage is not backed up by us, and at the end of your service the disks are securely erased, as described in data deletion. Backups are your responsibility: copy checkpoints, datasets you cannot rebuild and configuration to storage you control. To move data in and out, see rsync and scp.

rsync over SSH mirrors a folder to another machine of yours:

rsync -a --info=progress2 /data/checkpoints/ backup@203.0.113.20:/backups/gpu1/checkpoints/

restic keeps encrypted, deduplicated snapshots, here on an SFTP server you run:

sudo apt install -y restic
restic -r sftp:backup@203.0.113.20:/srv/restic/gpu1 init
restic -r sftp:backup@203.0.113.20:/srv/restic/gpu1 backup /data/checkpoints
restic -r sftp:backup@203.0.113.20:/srv/restic/gpu1 snapshots
restic -r sftp:backup@203.0.113.20:/srv/restic/gpu1 restore latest --target /tmp/restore-test

restic asks for a repository password; store it safely, because the backup cannot be read without it. Schedule backups with cron or a systemd timer, and test a restore, as in the last command, before you need one.

Troubleshooting

• Device or resource busy from mkfs or mdadm. The drive is mounted or already part of an array. Check lsblk and cat /proc/mdstat.
• The array comes back as /dev/md127 after a reboot. It was not saved in mdadm.conf; repeat step 3 of the RAID section. The fstab entry keeps working because it uses the filesystem UUID.
• A mount fails at boot. Thanks to nofail, the server still boots. Log in and check sudo findmnt --verify and sudo journalctl -b | grep -i mount.