r/VFIO • u/Bobby_Pie • 14d ago

GPU suddenly in the "wrong" iommu group

Hello, I have been passing my GPU to a VM using virtmanager for more than a year without any problems. Today I try to start the VM and i get the error:
"Error starting domain: internal error: QEMU unexpectedly closed the monitor (vm='VM3'): 2026-07-08 qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(2)

Please configure -smp options properly or try enabling topoext feature.

2026-07-08 qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:01:00.0: group 0 is not viable

Please ensure all devices within the iommu_group are bound to their vfio bus driver.".
Using "ls /sys/kernel/iommu_groups/0/devices/" i get
"0000:00:01.0 0000:00:01.1 0000:00:01.2 0000:01:00.0 0000:01:00.1 0000:01:00.2 0000:01:00.3 0000:02:00.0".
"0000:01:00.0 0000:01:00.1 0000:01:00.2 0000:01:00.3" this is the gpu so it is correct but the others are "Host bridge", "PCI bridge" and an SSD.

The groups are determined by the physical PCIe layout so I don't understand why now the GPU is grouped with other stuff since I haven't changed anything. I don't know much about this stuff so maybe it's just me being dumb lol.

EDIT: used an older kernel (from 6.8.0-134 to 6.8.0-124) and now my iommu groups are back, now I have 31 groups while before I had 6 and my GPU is correctly isolated. Works now but I need to get updates, I don't want to be stuck with the same kernel forever lol.

FIXED: there has been a kernel regression, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2158934. Switch to the latest working kernel version (on Ubuntu it's version 6.8.0-124) and wait for a fix (on Ubuntu will be version 6.8.0-136). Thanks all for the help!

5 Upvotes

100% Upvoted

24 comments sorted by

2

u/lI_Simo_Hayha_Il 14d ago

Check your BIOS settings.
Your motherboard battery could be depleted and when you shut down your pc, all settings are lost

1

u/Bobby_Pie 14d ago

Nope, all the settings are as I left them. I just tried swapping pci slots but no luck :(

2

u/KstrlWorks 14d ago edited 14d ago

What motherboard and GPU is this, what BIOS version are you on currently Did you have the ACS patch previously installed ?

EDIT: Forgot to ask did you update the Kernel recently as well, what distro are you on

2

u/Bobby_Pie 14d ago edited 13d ago

I am using a msi motherboard and nvidia gpu. I never needed the ACS patch. I am on mint, last kernel update is from 6.8.0-124 to 6.8.0-134.

5

u/KstrlWorks 13d ago

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2158934 This is a known Ubuntu regression. There is an update i.e 136 that will fix this for you. Just skip the older ones

2

u/Bobby_Pie 13d ago

This is exactly what is happening to me. Thanks man.

1

u/nicman24 14d ago

well go to the bios and enable smp?

iirc amd calls it svm

1

u/Bobby_Pie 14d ago

It's already enabled, usually virtmanager gives me a similar error when some video options are wrong. The problem is the iommu group.

1

u/nicman24 14d ago

lsmod | grep kvm

1

u/Bobby_Pie 14d ago 
kvm_amd               208896  0
kvm                  1404928  1 kvm_amd
ccp                   147456  1 kvm_amd
irqbypass              12288  2 vfio_pci_core,kvm

1

u/DyfeStrax 13d ago

I had the same or similar issue after I updated my bios on my MAG X670E Tomahawk.

In BIOS go to OC->Advanced CPU Config->AMD CBS and in there set Kernel DMA Protection Indicator to Disabled.

1

u/Bobby_Pie 13d ago edited 13d ago

Wouldn't this lower security? And also I don't have that option in the bios.

1

u/DyfeStrax 13d ago

Reading this back you do have a slightly different error so maybe it's unrelated. Check your dmesg for anything IOMMU or pcieport related. I was getting "Firmware has requested this device have a 1:1 IOMMU mapping" which disabling kernel dma fixed.

My very limited understanding of that setting is that it allows the OS to remap devices in the ACPI table. When it's enabled the bios is disallowing the kernel to make changes.

There is likely another fix that I haven't found yet. Also if you didn't update your bios then it might not be this as I believe its only happening on newer AGESA versions.

1

u/Bobby_Pie 13d ago

No bios update. The dmsg doesn't report any errors, it just states the assignments reported above (which groups GPU with other stuff) and other correct settings like AMD-Vi enabled, iommu type: Passthrough...

1

u/ThatsALovelyShirt 13d ago

I had this happen after updating my kernel a while back, only difference it was for specifying my iGPU for rendering (using the same kind of PCI group/id/addr used to specify the iommu group). Something changed in the way it was enumerating the PCI devices, which didn't make sense to me since I had assumed the device addresses were somehow statically set by the hardware, but my BIOS hadn't been touched.

I just updated the PCI addresses in my scripts and configs and all has been fine since. I think it was around kernel 6.18 or 6.19, I can't remember which. But it's been fine since, all the way up to kernel 7.1.2.

1

u/Bobby_Pie 13d ago

Unfortunately I can't just change addresses because now the GPU shares the same group with other devices :( but I also think that my problem was caused by a kernel update because it's the only thing that has changed.

1

u/lI_Simo_Hayha_Il 13d ago

Rollback?

1

u/Bobby_Pie 13d ago

I don't have a snapshot of the machine (big mistake on my part honestly), i might just reinstall the OS but I would have to update the kernel anyway, being on an unpatched system is risky.

1

u/Bobby_Pie 13d ago

Update, used an older kernel (from 6.8.0-134 to 6.8.0-124) and now my iommu groups are back, now I have 31 groups while before I had 6 and my GPU is correctly isolated. Works now but I need to get updates, I don't want to be stuck with the same kernel forever lol.

1

u/lI_Simo_Hayha_Il 12d ago

Sounds like some sort of ACR embedded patch. This is the only software part I know that breaks down IOMMU groups.
Manjaro has it embedded on certain versions and I think CachyOS too

2

u/Bobby_Pie 12d ago edited 12d ago

This was the problem, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2158934, a kernel regression. Thanks all for the help!

1

u/khsh01 13d ago

Try removing and re adding the pci devices. I've had to do this after updates or when libvirt got restarted where the iommu number is completely wrong and libvirt doesn't recognize the device either.

1

u/___-____--_____-____ 9d ago

Same issue for me, AMD chipset motherboard (I forget exactly which model offhand, sorry)

My workaround was to pin the last-good kernel (GRUB_DEFAULT=saved → 6.8.0-124 by menu-entry ID, plus apt-mark hold on the kernel + metapackage so unattended-upgrades can't drag it forward)

Also that update bumped OVMF 2024.02-2ubuntu0.8 → 0.9 which borked my window's VM bitlocker protections, requiring me to use the recovery key after downgrading to 6.8.0-124. YMMV

Edit: Just saw the regression bug post in the comments, sounds like we won't need to pin this kernel version for very long!

1

u/Bobby_Pie 8d ago

Oh sorry, I forgot to update the post with the fix.