Starting out my server setup, I needed to get GPU passthrough working for NVR/Jellyfin. Unfortunately something I didn’t know about the HP server when I bought it was some of it’s…quirks.

Initial setup and configuration

I started out going through the normal process of passing through PCIe device to VM.

Eventually ending up adding

quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off

to my kernel boot arguments

Make sure vfio modules get loaded by adding to /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Finding vendor/device IDs with

root@localhost:~# lspci -v | grep VGA
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH (prog-if 00 [VGA controller])
07:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P600] (rev a1) (prog-if 00 [VGA controller])
lspci -n -s 07:00
root@localhost:~# lspci -n -s 07:00
07:00.0 0300: 10de:1cb2 (rev a1)
07:00.1 0403: 10de:0fb9 (rev a1)

Replacing 07:00 with the appropriate address found with the first lspci command And adding vendor/device IDs into /etc/modprobe.d/vfio.conf with

options vfio-pci ids=10de:1cb2,10de:0fb9 disable_vga=1

Blacklist gpu drivers

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Also added to /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

Update our initramfs

update-initramfs -u -k all

Reboot and make sure everything in BIOS is set appropriately, IOMMU/Vt-d enabled etc. Adding the GPU to the VM and trying to start now leads an IOMMU permission error of some sort Failed to set iommu for container: Operation not permitted

HPE and hp-scripting-tools

Thanks to level1techs.com forums I found the hp-scripting-tools from HPE….

What’s happening?

HP has some shared memory features that seem to block the PCIe slots, thankfully we can exclude them.

Exclude from Reserved Memory Region

Download conrep_rmrds.xml

create exclude.dat file with the contents

<Conrep> <Section name="RMRDS_SlotX" helptext=".">Endpoints_Excluded</Section> </Conrep>

replacing RMRDS_SlotX with the appropriate slot of your PCIe device, in my case of full sized GPU, it’s in slot 2.

execute conrep -l -x conrep_rmrds.xml -f exclude.dat

verify with conrep -s -x conrep_rmrds.xml -f verify.dat

PCIe slot should be listed as Excluded now.

We’ve now disabled RMRR for our PCIe slot containing our GPU

Reboot

You should now be able to startup VM with GPU passed to it.