Proxmox PCIe Passthrough for pfSense NIC

So you're getting hardcore about your network setup, and can't sleep at night knowing your proxmox host where you are running pfSense is addressable from your WAN network. PCIe passthrough is the solution! Fair warning though, this is a little tricky and dependant on your hardware. If you follow this path, you may end up updating your BIOS or locking yourself out of your hardware remotely. There are other options to get to a similar end-goal, however these are either less secure or less performant depending on your hardware.

  1. You could set forward delay and aging time to 0 on the Linux Bridge, making it act like a conventional hub not a switch.
  2. You could use OVS Bridges and the mirroring feature supported by OpenVSSwitch. For me, I decided I'd just go all the way and use PCIe passthrough as it is the most secure and performant. Proxmox has a guide on how to preform PCIe passthrough: https://pve.proxmox.com/wiki/Pci_passthrough However I ran into some issues along the way, so here's my synthesis of it! 1

TLDR; Guide

  1. Enable IOMMU on your host. This will varry in procedure depending on your CPU Architecture. If you don't have an intel processer, continue here. Otherwise: a. Edit /etc/default/grub b. Change line GRUB_CMDLINE_LINUX_DEFAULT to be GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" c. Edit /etc/modules and add:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

d. Execute update-grub and reboot the system e. To validate, execute dmesg | grep -e DMAR -e IOMMU f. If you're unlucky and have a system that isn't new enough to support the interrupts, you may need to enable unsafe interrupt with: echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

  1. Shutdown your pfSense VM
  2. Ensure your pfSense machine type is Q35
  3. On your pfSense vm from Proxmox UI, click Hardware->Add->PCI Device a. Clicking the Device dropdown should populate a list of PCI devices on your system. If this doesn't exist, then IOMMU is not enabled yet on your system or the mappings weren't correctly generated. b. Add your required Ethernet device. Be careful here! The one you select will render all interfaces on that same group unusable from your host. I.E. If you have one NIC with two Ethernet ports and bridge that to pfSense, you'll lose the ability to use either of those Ethernet ports on your Proxmox host, rendering yourself unable to connect to it remotely.
  4. Boot pfSense again, re-configure the interfaces and you're done!