TL;DR: on 14 May 2026, my Docker VM froze. Two saturations at once: RAM was thrashing into swap, and Proxmox's LVM thin pool sat at 97.8% — dmeventd: Insufficient free space, every I/O blocked. The 250 GB NVMe was full and the volume group had zero free bytes left: no room to repair. The real fix was one sentence — a bigger disk — and one hardware problem: the motherboard has a single M.2 slot. Here is how I migrated a Proxmox host onto an NVMe four times larger with dd, a USB relay drive and zero reinstall. Plus the discard=on lesson I only understood a week later.

A homelab works fine right up until the day it doesn't work at all. Mine runs on Proxmox: two virtual machines — an Unraid array and a Docker VM hosting around sixty containers (Plex, Immich, Paperless, and all the rest). For months, I never looked at the disk space. That was the mistake.

This post is the story of an outage, then of a system-disk migration carried out from a live USB, step by step. If you run Proxmox on an NVMe that is filling up, the epilogue concerns you directly.

One Sunday, VM 101 freezes

The Docker VM — 101 in Proxmox — stopped responding. The UI dead, the containers unreachable. The diagnosis revealed not one failure but two, at once.

RAM first. The VM was using 96% of its memory — 26 GiB. At that level the guest kernel fell back on swap, then spent all its time shuttling pages between RAM and disk. The swap thrashing triggered CPU soft-lockups: the processor waits so long on an I/O that it declares itself stuck. The VM was frozen alive.

Storage next — and this one is worse. Proxmox keeps its virtual disks in an LVM thin pool named pve/data. It was 97.8% full. At that threshold dmeventd — the daemon watching LVM volumes — returned its verdict: Insufficient free space. Every write to the pool was blocked. A VM whose disk can no longer write anything does not do much.

A thin pool is a reservoir of blocks. The virtual disks — here 32 GB for the Unraid array, 150 GB for the Docker VM — are thin volumes: they advertise their maximum size but only draw from the reservoir the blocks actually written. Handy for over-committing… until the day the reservoir genuinely runs dry. And then it is not the volumes that overflow: it is the pool.

Emergency triage calmed the fire without putting it out. I lowered the memory limits of the greediest containers, cut swap from 8 to 2 GiB, deleted a ghost snapshot stuck in place, and used the few gigabytes reclaimed to extend the thin pool to 159 GB. The VMs came back. But the pool still read 93.5%. I had pushed the problem back a few days, not solved anything.

The real problem: there was no margin left

Why couldn't I just grow the pool? Because there was nothing left to grow it with.

Here is what the NVMe looked like before the migration:

/dev/nvme0n1 — 232.9 GiB (Crucial CT250P5SSD8)
├── nvme0n1p1    1007K   BIOS boot
├── nvme0n1p2       1G   EFI System
└── nvme0n1p3   231.9G   Linux LVM (VG: pve)
    ├── pve-swap       2G
    ├── pve-root      68G   ext4 (15 GiB used)
    ├── pve-data   159.1G   thin pool (93.5%)
    │   ├── vm-100-disk-0   32G
    │   └── vm-101-disk-0  150G
    └── (free: 0G)

Look at the last line. Volume group pve had 0 GB free. Every byte of the disk had been handed out: 2 GB of swap, 68 GB of root, and all the rest poured into the thin pool. The "me" who had installed this Proxmox months earlier had ticked the default option and allocated everything. Not one byte left to breathe.

That is exactly the margin you miss on the day of an incident. No free space in the VG, no way to extend the pool; a full 250 GB disk, no bigger disk. Every door was shut but one: replace the NVMe with a larger model. A 1 TB Crucial P5 — four times the room.

A single M.2 slot — the constraint that complicates everything

Replacing a system disk, on paper, is trivial: plug the new one in next to the old, clone across, unplug the old. Except this server's motherboard — a Gigabyte B450M DS3H — has only one M.2 slot. No way to have the old and the new NVMe connected at the same time. No way to clone disk to disk directly.

The workaround: a relay drive. Clone the NVMe to an external disk, power down, physically swap the two NVMe drives, power back up, and restore the clone from the external disk onto the new NVMe. Two passes instead of one, but it sidesteps the constraint. I had a 12 TB WD Elements on USB sitting empty — a perfect relay.

NVMe 250 GB source — full WD Elements 12 TB USB relay NVMe 1 TB target — new 1. dd — clone 2. bounded dd Between the two passes: power off, pull the 250 GB drive, fit the 1 TB into the only M.2 slot.

There is a faster variant: slip the new 1 TB NVMe into a USB-to-M.2 enclosure (about ten euros), clone straight from internal NVMe to external NVMe in a single pass at SSD speed, then physically swap the two drives. No relay drive, no double copy. I didn't do it — I already had the WD Elements — but if you are planning this kind of operation, the enclosure is well worth the price.

Clone the disk, don't reinstall

I could have reinstalled Proxmox fresh on the 1 TB and reconfigured everything. I didn't, and that was deliberate.

A block-for-block clone with dd copies everything: the partition table, the EFI partition, the GRUB bootloader, and the LVM physical volume with its UUIDs. After the restore, the new disk is a perfect twin of the old one. Proxmox boots, the VMs boot, the network config, the users, the certificates — nothing moved, because as far as the system is concerned nothing changed. We only laid the same bytes onto a bigger disk.

The price: the VMs are shut down for the whole operation (a cold clone, no live migration), and it takes two dd passes across USB. In my case, about 17 minutes per pass at 235 MB/s over USB 3. An afternoon of downtime, schedulable.

The procedure, step by step

All the surgery runs from a live USB. Not the Proxmox installer ISO: its rescue environment is too minimal — no lsblk, no parted, /dev/disk/by-id/ not populated. I used SystemRescue, which carries the full disk toolkit and drops you straight into a root shell.

Step 1 — Shut the VMs down cleanly

From the Proxmox UI, before touching any hardware:

qm shutdown 101 --timeout 120
qm shutdown 100 --timeout 120
qm list

Step 2 — Boot into SystemRescue

With the SystemRescue USB stick and the relay drive plugged in, reboot, enter the BIOS (the DEL key on Gigabyte) and boot from the stick. SystemRescue drops you straight into a root shell. For a French keyboard: loadkeys fr.

Step 3 — Identify the disks

The most dangerous step in the whole procedure. The /dev/sdX letters change from one boot to the next — never trust them. Address the USB disk by its stable path under /dev/disk/by-id/; the NVMe, the only one in the machine, keeps its fixed name /dev/nvme0n1.

lsblk -d -o NAME,SIZE,MODEL,SERIAL
ls -l /dev/disk/by-id/ | grep -E 'nvme|usb'

On this server, the inventory held a surprise: with Proxmox stopped, the eight 4 TB disks of the Unraid array (VM 100) showed up as bare disks, sda through sdh. Catching one of them in a dd command would have pulverized the NAS. Hence the iron rule: define named variables, and never type a raw sdX.

SRC=/dev/nvme0n1
DST=/dev/disk/by-id/usb-WD_Elements_XXXX-0:0
lsblk "$SRC" "$DST"
# $SRC must show ~232G (nvme0n1p1/p2/p3) — $DST must show ~10.9T.
# If the sizes look swapped: STOP, run nothing.

Step 4 — Clone the NVMe onto the relay drive

dd if="$SRC" of="$DST" bs=4M conv=fsync status=progress

About 17 minutes at 236 MB/s. Then verify the clone — but comparing only the NVMe's size: the relay drive is far larger, so comparing both whole disks would be a guaranteed false failure.

SIZE=$(blockdev --getsize64 "$SRC")
echo "$SIZE"          # note this value — reused in step 6
cmp -n "$SIZE" "$SRC" "$DST" && echo "CLONE OK"

Step 5 — Physically swap the NVMe drives

Power off (poweroff), unplug the power. Remove the 250 GB Crucial — label it and set it aside: this is the full rollback — and fit the 1 TB into the same M.2 slot. The SystemRescue stick and the relay drive stay plugged in.

Step 6 — Restore the clone onto the new NVMe

Boot back into SystemRescue. The NVMe now reports ~931 GB. The restore goes the opposite direction from step 4 — and it must be bounded:

NVME=/dev/nvme0n1
DST=/dev/disk/by-id/usb-WD_Elements_XXXX-0:0
lsblk "$NVME" "$DST"

CLONE_SIZE=250059350016   # the size recorded in step 4
dd if="$DST" of="$NVME" bs=4M iflag=count_bytes count="$CLONE_SIZE" conv=fsync status=progress
sync
cmp -n "$CLONE_SIZE" "$NVME" "$DST" && echo "RESTORE OK"

Without the count=, dd would copy the entire 12 TB relay drive and overflow the NVMe long before it finished — hundreds of gigabytes for nothing.

Step 7 — Repair the GPT, extend the partition and the pool

The clone is 233 GB; laid onto a 1 TB disk, its backup GPT table ends up in the wrong place and hides the free space. We relocate it, then — the crucial point — unplug the relay drive before any LVM command: because the clone is bit-for-bit, the relay and the NVMe carry the same physical-volume UUID. As long as both are connected, LVM sees a duplicate PV and may operate on the wrong disk.

vgchange -an pve
sgdisk -e "$NVME"            # relocate the backup GPT to the end of the disk

# >>> physically unplug the relay drive's USB cable <<<
pvscan --cache
pvs                          # one single PV for VG "pve", no "duplicate" warning

growpart "$NVME" 3           # grow partition 3 to all available space
partprobe "$NVME"
pvresize /dev/nvme0n1p3
vgchange -ay pve

lvextend --poolmetadatasize +1G pve/data   # the pool quintuples — metadata must keep up
lvextend -l +95%FREE pve/data              # +95%, never +100%: keep a margin
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
vgs pve

Note: parted may be missing from the PATH on SystemRescue; growpart does the job just as well, and handles the GPT relocation itself.

Step 8 — Reboot

Remove the SystemRescue stick and reboot. Since dd copied the EFI partition, GRUB and every UUID identically, the server boots on the new NVMe without ever knowing about the change.

Step 9 — Verify

pveversion
qm list
qm status 100
qm status 101

If the VMs have the onboot option, they are already back up on their own.

One detail that may worry you: proxmox-boot-tool status returns E: /etc/kernel/proxmox-boot-uuids does not exist. That is normal on a classic LVM + GRUB install — proxmox-boot-tool only matters for ZFS or systemd-boot setups. A successful boot is proof enough that the boot chain is intact.

The traps that turn a migration into a disaster

The first draft of this procedure did not hold together. It was the review, then the field test, that hardened it. Here are the eight traps that make the difference between a clean migration and a wiped disk.

  • Verify the clone against the right size. A cmp over both whole devices is a guaranteed false failure — the relay drive is larger. Use cmp -n with the clone's exact byte count.
  • Bound the restore with count=. The 12 TB relay is far bigger than the 250 GB clone; an unbounded dd copies the entire relay until it overflows the NVMe.
  • Unplug the relay drive before any LVM command. A bit-for-bit clone means an identical PV UUID. LVM sees a duplicate PV and may run pvresize/lvextend on the wrong disk.
  • Relocate the backup GPT. The clone is 233 GB; on a 1 TB disk its backup GPT table lands in the wrong place and partition growth can't see the free space. sgdisk -e puts it back.
  • Address USB disks by /dev/disk/by-id/. The sdX letters change between boots. The NVMe, the only one in the machine, keeps its stable /dev/nvme0n1.
  • Know your live USB. parted may be missing from the PATH on SystemRescue; growpart or sgdisk do the job. And the Proxmox installer ISO is too bare for disk surgery.
  • Leave a margin in the volume group. Never lvextend -l +100%FREE. That "0 GB free" is exactly what blocked the incident response. I left around 5%.
  • Inventory every disk before the first dd. With Proxmox stopped, the eight 4 TB disks of the Unraid array showed up as bare disks. Catching one in a dd would have destroyed the NAS. Named variables, never a raw sdX.

The result

The server booted on the new NVMe on the first try. Here is the final layout:

/dev/nvme0n1 — 931.5 GB (Crucial CT1000P5SSD8)
├── nvme0n1p1    1007K   BIOS boot
├── nvme0n1p2       1G   EFI System
└── nvme0n1p3  ~930.5G   Linux LVM (VG: pve, VFree 34.83 GiB)
    ├── pve-swap        2G
    ├── pve-root      ~68G
    ├── pve-data   820.91G   thin pool (18.12%)
    │   ├── vm-100-disk-0   32G
    │   └── vm-101-disk-0  150G
    └── (VG free: 34.83G — safety margin)
  • LVM thin pool — from 159 GB jammed at 93.5%, to 820.91 GB used at just 18.12%.
  • Pool metadata — extended from 1.40 to 2.40 GB, to keep up with a pool five times larger.
  • Volume group free space — from 0 to 34.83 GB. That is the safety margin that was missing on 14 May.
  • Boot — Proxmox VE 9.1.1 (kernel 6.17.2-1-pve) came back without a hitch, both VMs restarted automatically by onboot.

No reconfiguration. Because dd copied EFI, GRUB and the UUIDs identically, the system simply never noticed it had changed disks.

Epilogue — the "99%" that wasn't

When closing the ticket, I had noted a follow-up item: VM 101's virtual disk appeared "99% full". A week later, when I went to enlarge it, I realized I had read the wrong metric.

That "99%" was not the fill level of the guest filesystem. It was the Data% of the thin volume vm-101-disk-0 — the share of the reservoir that this volume had consumed. Inside the VM, the / filesystem was actually at 72%: 99 GB used out of 146, 41 GB free. No enlargement was needed. But then, why had the thin allocation ballooned to 99%?

The answer is one word: discard. The VM's scsi0 disk did not have the discard=on option. Without it, the guest's TRIM commands — which signal freed blocks — never reached the thin pool. The guest could run its weekly fstrim all it wanted, the pool never reclaimed those blocks. The allocation could only grow, never shrink. A disk that empties out on the guest side but stays "full" on the pool side: that is the trap of thin provisioning without discard.

The fix:

qm set 101 --scsi0 local-lvm:vm-101-disk-0,size=150G,discard=on,ssd=1

A reboot of the VM (the discard option is set when the block device is created), then an fstrim -av inside the guest — and 48 GB are handed back to the pool in one go. The Data% of vm-101-disk-0 drops from 99.17% to 70.46%, the thin pool from 18.12% to 12.88%. The weekly fstrim.timer now keeps the allocation tidy on its own.

And the last missing link: I had no automated backups. The 12 TB WD Elements that had served as the relay was repurposed as a backup disk — wiped, reformatted to ext4, mounted at /mnt/wd-backup, declared as Proxmox storage. A weekly vzdump job now backs up both VMs every Sunday at 2 a.m., in snapshot mode, with zstd compression. The 14 May incident could have cost around sixty containers and their data. Next time, there will be a net.

What I'm taking away

  • Never allocate 100% of a volume group. Keep around 5% free. That margin is exactly what lets you repair an incident instead of suffering it.
  • Thin provisioning requires discard=on end to end. Without it, the pool's allocation only climbs, even when the guests empty out. TRIM must be able to travel the whole chain.
  • Watch the thin pool, not just the disks. A guest volume at 72% and a pool at 97% tell two different stories — and it is the pool that blocks your I/O.
  • A dd clone is an underrated migration tool. It carries EFI, GRUB and the LVM UUIDs identically: the system never sees the difference.
  • Know your live USB before you need it. Which tools are present, how the disks are named, what the rescue ISO actually carries.
  • No backup, no net. A homelab without a scheduled vzdump is not a homelab, it is a bet.

The 250 GB NVMe is labeled and stored away. It stays intact for a week or two — long enough to confirm the new disk is stable — as a rollback path. Because a migration is never truly finished until you have a way to undo it.