Hi,
I'm aiming to run few KVM virtual machines on RPi5-8GB, I can create them, start them, install OS, all works quite OK, but for one of them I would like to pass to it NVME disk that I have connected via NVME hat. This disk works fine, I was using it for quite some time. My thinking is that instead of emulating some IDE/SCSI controller(where might be some performance loss due to emulation), I can add PCI device as pass-through.
I have added it like:
But when I start that KVM virtual machine it fails with:
And indeed iommu_group does not exists:I was thinking that probably something is missing in official RPI kernel, so I have cloned latest from GitHub(6.11.0-rc2-v8-16k), in configuration I have enabled absolutely everything containing IOMMU and VFIO, compiled and booted into it but still can't really get it. When I check dmesg for IOMMU I can see following:
Strange is that in those 2 IOMMU groups, there are not a single PCI device and even lspci -vv does not show any IOMMU details for any PCI device.
So my question would be: Is there any hardware limitation and this would not work at all? I'm I missing some boot param? Is it something that is not yet implemented in kernel(bcm2712-iommu driver)?
I'm aiming to run few KVM virtual machines on RPi5-8GB, I can create them, start them, install OS, all works quite OK, but for one of them I would like to pass to it NVME disk that I have connected via NVME hat. This disk works fine, I was using it for quite some time. My thinking is that instead of emulating some IDE/SCSI controller(where might be some performance loss due to emulation), I can add PCI device as pass-through.
I have added it like:
Code:
<disk type='nvme' device='disk'> <driver name='qemu' type='raw'/> <source type='pci' managed='yes' namespace='1'> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </disk>But when I start that KVM virtual machine it fails with:
Code:
error: internal error: Invalid device 0000:01:00.0 iommu_group file /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlinkAnd indeed iommu_group does not exists:
Code:
root@rpi:/home/lukas# ls -a /sys/bus/pci/devices/0000\:01\:00.0/. ari_enabled current_link_speed driver local_cpulist msi_bus power_state resource subsystem_device.. broken_parity_status current_link_width driver_override local_cpus msi_irqs remove resource0 subsystem_vendoraer_dev_correctable class device enable max_link_speed nvme rescan resource4 ueventaer_dev_fatal config devspec irq max_link_width pools reset revision vendoraer_dev_nonfatal consistent_dma_mask_bits dma_mask_bits link modalias power reset_method subsystemroot@rpi:/home/lukas#Code:
root@rpi:/home/lukas# dmesg | grep -i IOMMu[ 0.008903] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL **[ 0.008913] ** IOMMU data structures, which may compromise security on **[ 0.909155] iommu: Default domain type: Translated[ 0.913976] iommu: DMA domain TLB invalidation policy: strict mode[ 1.463232] bcm2712-iommu-cache 1000005b00.iommuc: bcm2712_iommu_cache_probe[ 2.820136] bcm2712-iommu 1000005100.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774[ 2.828469] bcm2712-iommu 1000005100.iommu: bcm2712_iommu_probe: Success[ 2.835709] bcm2712-iommu 1000005200.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774[ 2.844033] bcm2712-iommu 1000005200.iommu: bcm2712_iommu_probe: Success[ 2.851175] bcm2712-iommu 1000005280.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774[ 2.859496] bcm2712-iommu 1000005280.iommu: bcm2712_iommu_probe: Success[ 6.338004] pispbe 1000880000.pisp_be: bcm2712_iommu_of_xlate: MMU 1000005100.iommu[ 6.338015] pispbe 1000880000.pisp_be: bcm2712_iommu_probe_device: MMU 1000005100.iommu[ 6.338744] pispbe 1000880000.pisp_be: bcm2712_iommu_device_group: MMU 1000005100.iommu[ 6.338765] pispbe 1000880000.pisp_be: Adding to iommu group 0[ 6.338772] pispbe 1000880000.pisp_be: bcm2712_iommu_attach_dev: MMU 1000005100.iommu[ 6.339730] rpivid 1000800000.codec: bcm2712_iommu_of_xlate: MMU 1000005100.iommu[ 6.339736] rpivid 1000800000.codec: bcm2712_iommu_probe_device: MMU 1000005100.iommu[ 6.339744] rpivid 1000800000.codec: bcm2712_iommu_device_group: MMU 1000005100.iommu[ 6.339750] rpivid 1000800000.codec: Adding to iommu group 0[ 6.339753] rpivid 1000800000.codec: bcm2712_iommu_attach_dev: MMU 1000005100.iommu[ 6.419015] vc4-drm axi:gpu: bcm2712_iommu_of_xlate: MMU 1000005200.iommu[ 6.419025] vc4-drm axi:gpu: bcm2712_iommu_probe_device: MMU 1000005200.iommu[ 6.419032] vc4-drm axi:gpu: bcm2712_iommu_device_group: MMU 1000005200.iommu[ 6.419038] vc4-drm axi:gpu: Adding to iommu group 1[ 6.419044] vc4-drm axi:gpu: bcm2712_iommu_attach_dev: MMU 1000005200.iommuroot@rpi:/home/lukas#So my question would be: Is there any hardware limitation and this would not work at all? I'm I missing some boot param? Is it something that is not yet implemented in kernel(bcm2712-iommu driver)?
Statistics: Posted by coRp — Mon Aug 05, 2024 5:46 pm — Replies 0 — Views 23