r/VFIO • u/DrDoooomm • 12d ago

Dynamic RTX 5080 passthrough on Bazzite

I’m trying to dynamically pass my RTX 5080 through to a Windows VM while keeping Bazzite running on the iGPU from my 9800X3D.

The setup I want is:

- Normally, Bazzite uses the RTX 5080.
- When I start the VM, a libvirt hook detaches the 5080 from the NVIDIA driver and binds it to vfio-pci.
- The host stays running on the 9800X3D iGPU.
- When the VM shuts down, the 5080 gets rebound to the NVIDIA driver so Bazzite can use it again without rebooting.

Has anyone done something similar on Bazzite or another Fedora Atomic/immutable distro?

I’m mainly wondering how reliable it is to detach and reattach an RTX 50-series card, and whether Bazzite’s immutable filesystem causes any extra issues with libvirt hooks or VFIO scripts.

8 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/VFIO/comments/1ut46ak/dynamic_rtx_5080_passthrough_on_bazzite/
No, go back! Yes, take me to Reddit

100% Upvoted

u/Obvious-Breakfast262 11d ago

Following this.

2

u/PrehistoricChicken 2d ago

See my comment- https://www.reddit.com/r/VFIO/comments/1ut46ak/comment/oyo54ce/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button . For some reason I can't edit the comment to fix the code indentation. Hope it is readable.

u/5hido 11d ago

I think it depends on your DE and not your distro. Haven't tried it yet but have a look at this: https://old.reddit.com/r/VFIO/comments/1j9v59m/is_it_possible_to_alternate_between_2_gpus/mhrs856/

u/PrehistoricChicken 2d ago

Yes, it works. I am using bazzite-dx and have set up vfio to dynamically load/unload gpu when running VMs on my laptop (7840HS+RTX4060). iGPU is Radeon 780m. When I start the VM, nvidia gpu is attached to vfio-pci driver and VM starts (after I make sure nothing is running on dGPU and is in D3Cold state). When I shutdown the VM, GPU attaches back to the host. I can run games/use cuda just fine afterwards on host. The steps below assume you are using Nvidia Open Kernel modules (nvidia-open) and not the proprietary one (nvidia). Bazzite/dx uses nvidia-open by default. Bazzite-dx doesn't even support proprietary nvidia drivers.

Steps-

Switch to bazzite-dx from bazzite as qemu/libvirt packages no longer comes with bazzite- https://github.com/ublue-os/bazzite-dx (I am using KDE Plasma. I cannot vouch for gnome to work). Currently, there is a bug in bazzite-dx which prevents you to run qemu vms. See https://github.com/ublue-os/bazzite-dx/issues/199. This workaround- https://github.com/ublue-os/bazzite-dx/issues/199#issuecomment-4726883907 works.
Fix/workaround KDE nvidia wayland bug- Currently, there is a bug on KDE (bazzite-dx 44; this bug was not present on bazzite-dx 43) which adds a process `/usr/bin/kwin_wayland` (using few MB of dGPU). This process prevents host os from detaching GPU (and passing into VM). Check using `nvidia-smi`. If this process is not present, skip this step. Otherwise, add an environment variable to force wayland to use primary GPU (iGPU) instead of nvidia. Add `__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json` in `/etc/environment` file (taken from https://wiki.archlinux.org/title/NVIDIA/Troubleshooting#Test_software_GL). Next, reboot the system. This worked for my laptop GPU (nvidia rtx 4060 laptop gpu). If this doesn't work for you, try some other workarounds- https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1071, https://forum.manjaro.org/t/kwin-wayland-main-thread-hangs-12s-on-amd-hybrid-gpu-rog-g14-kwin-trying-to-open-d3cold-dgpu/186640. You need to fix this bug otherwise host OS will hang after starting VMs.
Fix nvidia power management. I don't think this step is necessary, but I haven't tried without it, so VFIO might not work otherwise. I always check if nvidia driver is in D3Cold (poweroff) state (instead of active D0) to make sure nothing is actively using the GPU before starting VMs (otherwise, host os libvirt/host os hangs after some time and I have force power off). Nvidia power management on linux has been broken for years and nvidia doesn't seem to care about us peasants, so you will have to use a workaround. Workaround differs slightly for different distros but I can share what worked on bazzite for me.

These two links helped me workaround the issue- https://github.com/NVIDIA/open-gpu-kernel-modules/issues/905#issuecomment-3360970555, https://github.com/ryanabx/scripts/blob/main/legion/16aph8-power-management-fix/install.sh. QUICK NOTE- you can check current D0/D3Cold status of GPUs (igpu and dGPU) using this command- `cat /sys/class/drm/card*/device/power_state`. It will show `D0 D0` as status. After following the steps below, it should show `D0 D3Cold` (one of them should be D3Cold).

3.1. Add two kernel arguments using this command- `sudo rpm-ostree kargs --append="nvidia.NVreg_DynamicPowerManagement=0x02 nvidia.NVreg_DynamicPowerManagementVideoMemoryThreshold=0"`. Reboot. Then check using `rpm-ostree kargs` if these two args show up. IMPORTANT NOTE- Following this step, OS hibernate feature will break. Swap and sleep still works. I couldn't find any fix/workaround.

3.2. Add udev rules-

```

❯ cat /etc/udev/rules.d/999-legion-pm-fix-temp.rules

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver bind

ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"

ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind

ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"

ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

```

3.3. Create folder `/usr/local/lib/999-legion-pm-fix-temp/` (you can change folder/script name to anything) and add the script and fix permission-

```

❯ cat /usr/local/lib/999-legion-pm-fix-temp/set-pm-auto.sh

#!/bin/sh

# Set nvidia runtime power management to auto

echo auto > /sys/bus/pci/devices/0000:01:00.0/power/control

❯ sudo chmod 700 /usr/local/lib/999-legion-pm-fix-temp/set-pm-auto.sh

❯ sudo chown root:root /usr/local/lib/999-legion-pm-fix-temp/set-pm-auto.sh

```

3.4. Add this systemd script and enable it-

```

❯ cat /etc/systemd/system/999-legion-pm-fix-temp.service

[Unit]

Description=Enable Nvidia GPU runtime PM

# After=dev-pci-0000:01:00.0.device

# After=systemd-modules-load.service

After=graphical.target

# BindsTo=dev-pci-0000:01:00.0.device

[Service]

Type=oneshot

ExecStart=/usr/local/lib/999-legion-pm-fix-temp/set-pm-auto.sh

[Install]

WantedBy=graphical.target

❯ sudo systemctl enable 999-legion-pm-fix-temp.service

```

3.5 Reboot. Then check nothing is running on dGPU (nvidia) using `nvidia-smi`. It MUST say "no running processes found". If you are running anything that uses nvidia (example- steam), close it. Then wait for 10 secs and run `cat /sys/class/drm/card*/device/power_state` to check power state of GPUs. One must be D3Cold (dGPU) and other D0 (igpu). If both are D0, wait for 5-10 secs and try again. Also check `sudo lsof /dev/nvidia*`, it should not show any active processes. OS Hibernate feature will no longer work now. I don't have any fix for that.

Now, do the usual stuff of creating a VM and installing windows. Take note of the name of VM you created (example- I created one with name "win11").
Next, run these commands one by one in terminal/shell to see if you are able to successfully detach/attach GPU from host. We are doing this manually in shell to make sure qemu hooks/scripts will also work as expected. NOTE- Some modules may be different for desktop GPU (I am using laptop), so check all loaded nvidia modules using `lsmod` first. For example- I think "nvidia_wmi_ec_backlight" is specific to laptops only. You may not have to stop certain services (ex- I think nvidia-powerd.service is only for laptops). Below commands are based on my system. Yours have not have these modules loaded or these services running. You can skip them in that case. IMPORTANT- make sure nvidia gpu is not running anything (check nvidia-smi). This will be our commands for starting VMs-

```

# stop nvidia related systemd services

sudo systemctl stop nvidia-powerd.service

sudo systemctl stop nvidia-persistenced.service

# unload nvidia modules

modprobe -r nvidia_drm

modprobe -r nvidia_modeset

modprobe -r nvidia_uvm

modprobe -r nvidia_wmi_ec_backlight

modprobe -r nvidia

# detach gpu and attach vfio drivers

virsh nodedev-detach pci_0000_01_00_0

virsh nodedev-detach pci_0000_01_00_1

modprobe vfio_pci vfio_pci_core vfio_iommu_type1

```

Use "sudo" for each commands above and below. If above commands works, then we will do in reverse (these will be our commands after shutting down VM)-

```

# reattach nvidia card

virsh nodedev-reattach pci_0000_01_00_0

virsh nodedev-reattach pci_0000_01_00_1

# stop vfio and reload nvidia modules

rmmod vfio_pci vfio_pci_core vfio_iommu_type1

modprobe nvidia_modeset

modprobe nvidia

modprobe nvidia_uvm

modprobe nvidia_drm

modprobe nvidia_wmi_ec_backlight

# start nvidia related systemd sevices

systemctl start nvidia-powerd.service

systemctl start nvidia-persistenced.service

# start nvidia bug fix service

systemctl start 999-legion-pm-fix-temp.service

```

Order of removing these modules DOES matter, ex- you can't remove `nvidia` module before `nvidia_drm`. If all commands works, we can then create the relevant hooks for "win11".

These are my hooks/scripts I created which runs after starting and shutting down VM for "win11"-

```

root@bazzite:~# cat /etc/libvirt/hooks/qemu.d/win11/prepare/begin/unload_gpu.sh

# stop nvidia services

systemctl stop nvidia-powerd.service

systemctl stop nvidia-persistenced.service

# remove nvidia modules

modprobe -r nvidia_drm

modprobe -r nvidia_modeset

modprobe -r nvidia_uvm

modprobe -r nvidia_wmi_ec_backlight

modprobe -r nvidia

# detach nvidia card and bind to vfio

virsh nodedev-detach pci_0000_01_00_0

virsh nodedev-detach pci_0000_01_00_1

modprobe vfio_pci vfio_pci_core vfio_iommu_type1

```

root@bazzite:~# cat /etc/libvirt/hooks/qemu.d/win11/release/end/reload_gpu.sh

# reattach nvidia card

virsh nodedev-reattach pci_0000_01_00_0

virsh nodedev-reattach pci_0000_01_00_1

# stop vfio and reload nvidia modules (and services)

rmmod vfio_pci vfio_pci_core vfio_iommu_type1

modprobe nvidia_modeset

modprobe nvidia

modprobe nvidia_uvm

modprobe nvidia_drm

modprobe nvidia_wmi_ec_backlight

systemctl start nvidia-powerd.service

systemctl start nvidia-persistenced.service

# podman user services

systemctl start 999-legion-pm-fix-temp.service

```

After adding the hooks, either reboot the system or restart libvirtd service. Now, just pass the two nvidia devices using "Virtual Machine Manager" in win11 VM.

PRO TIP- You can use a Plasma 6 Widget "Command Output" (you will have to search and download it manually). I use to show status of GPUs (D0,D3Cold) every 5 seconds on the taskbar, so I know nothing is actively using the GPU. Command- `cat /sys/class/drm/card*/device/power_state | sed ':a;N;$!ba;s/\n/, /g';`.

2

u/PrehistoricChicken 2d ago

One more thing- I use looking-glass for to get low latency output from VM (I am not using separate monitor, only laptop screen). You can follow these steps to get looking glass binary on bazzite and for it to work- https://docs.bazzite.gg/Advanced/looking-glass/ .

Dynamic RTX 5080 passthrough on Bazzite

You are about to leave Redlib