求助：pve直通nvme ssd给飞牛

对角线

<p>pve使用的是8.3-1系统，已经做SR-IOV虚拟化和PCI直通</p> <pre><code>root@pvenas:~# cat tt.sh

!/bin/bash

for d in /sys/kernel/iommu_groups//devices/; do n=${d#/iommu_groups/}; n=${n%%/} printf 'IOMMU Group %s ' "$n" lspci -nns "${d##/}" done root@pvenas:~# sh tt.sh IOMMU Group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 1 00:00.0 Host bridge [0600]: Intel Corporation 12th Gen Core Processor Host Bridge [8086:4650] (rev 05) IOMMU Group 10 00:1c.1 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 [8086:7ab9] (rev 11) IOMMU Group 11 00:1f.0 ISA bridge [0601]: Intel Corporation Z690 Chipset LPC/eSPI Controller [8086:7a84] (rev 11) IOMMU Group 11 00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11) IOMMU Group 11 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11) IOMMU Group 11 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11) IOMMU Group 12 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d] IOMMU Group 13 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] [144d:a80c] IOMMU Group 14 03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04) IOMMU Group 15 04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04) IOMMU Group 16 00:02.1 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 17 00:02.2 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 18 00:02.3 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 19 00:02.4 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 2 00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01) IOMMU Group 20 00:02.5 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 21 00:02.6 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 22 00:02.7 VGA compatible controller [0300]: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] [8086:4690] (rev 0c) IOMMU Group 3 00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller [8086:467f] IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11) IOMMU Group 4 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11) IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11) IOMMU Group 6 00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11) IOMMU Group 7 00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11) IOMMU Group 8 00:1b.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port [8086:7ac4] (rev 11) IOMMU Group 9 00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 [8086:7ab8] (rev 11) </code></pre> <p>IOMMU Group 13是nvme1n1，做的pve系统；IOMMU Group 12是nvme0n1需要直通给飞牛OS</p> <pre><code>root@pvenas:~# ls -la /sys/dev/block/ |grep -v loop |grep -v dm total 0 drwxr-xr-x 2 root root 0 Apr 4 20:45 . drwxr-xr-x 4 root root 0 Apr 4 20:45 .. lrwxrwxrwx 1 root root 0 Apr 4 20:45 259:0 -> ../../devices/pci0000:00/0000:00:1b.0/0000:02:00.0/nvme/nvme1/nvme1n1 lrwxrwxrwx 1 root root 0 Apr 4 20:45 259:1 -> ../../devices/pci0000:00/0000:00:1b.0/0000:02:00.0/nvme/nvme1/nvme1n1/nvme1n1p1 lrwxrwxrwx 1 root root 0 Apr 4 20:45 259:2 -> ../../devices/pci0000:00/0000:00:1b.0/0000:02:00.0/nvme/nvme1/nvme1n1/nvme1n1p2 lrwxrwxrwx 1 root root 0 Apr 4 20:45 259:3 -> ../../devices/pci0000:00/0000:00:1b.0/0000:02:00.0/nvme/nvme1/nvme1n1/nvme1n1p3 </code></pre> <p>从上面可看出nvme0n1的ids为144d:a80d，已经配置好vfio-pci，并在pve宿主机解绑nvme0n1</p> <pre><code>root@pvenas:~# lspci -nnk -d 144d:a80d 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d] Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a801] Kernel driver in use: vfio-pci Kernel modules: nvme </code></pre> <p>现在在虚拟机飞牛OS中添加PCI设备，也可直接看conf配置文件</p> <pre><code>root@pvenas:~# cat /etc/pve/qemu-server/101.conf bios: ovmf boot: order=scsi0;ide2;net0 cores: 8 cpu: host efidisk0: local:101/vm-101-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:01:00,pcie=1,device-id=0xa80d,sub-vendor-id=0x144d,sub-device-id=0xa801 hostpci1: 0000:00:02.2 ide2: local:iso/TRIM-0.8.43-795.iso,media=cdrom,size=2064512K machine: q35,viommu=intel memory: 16384 meta: creation-qemu=9.0.2,ctime=1743590704 name: fnOS net0: virtio=BC:24:11:64:D3:79,bridge=vmbr0,firewall=1 numa: 0 ostype: l26 scsi0: local:101/vm-101-disk-1.qcow2,iothread=1,size=128G scsihw: virtio-scsi-single smbios1: uuid=68c828fd-c785-4093-9d18-19b861fd9339 sockets: 1 vmgenid: 1acb4040-24dc-406b-876c-f35600e6be09 </code></pre> <p>或者在虚拟机飞牛OS的硬件中添加PCI设备</p> <p><img src="data/attachment/forum/202504/04/213454p977j5imsmw4b0zz.png" alt="101conf.png" title="101conf.png" /></p> <p>开启101虚拟机（飞牛OS）， 使用fdisk命令并没有看到直通的nvme ssd</p> <pre><code>root@wuyfnnas:~# fdisk -l Disk /dev/sda: 128 GiB, 137438953472 bytes, 268435456 sectors Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xbf393734

Device Boot Start End Sectors Size Id Type /dev/sda1 2048 194559 192512 94M 83 Linux /dev/sda2 194560 268435455 268240896 127.9G 83 Linux </code></pre> <p>但在101虚拟机中用lspci命令又能看到直通的nvme设备</p> <pre><code>root@wuyfnnas:~# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Device 1234:1111 (rev 02) 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) 00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03) 00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) 05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon 06:11.0 VGA compatible controller: Intel Corporation Alder Lake-S GT1 [UHD Graphics 770] (rev 0c) 06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device 09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI </code></pre> <p>就是01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)</p> <p>在飞牛OS中nvme 模块已经加载，而且内核在启动时识别了直通的 NVMe 控制器（显示为 nvme0: pci function 0000:01:00.0），但系统并没有创建对应的 NVMe 块设备（如 /dev/nvme0n1），是不能建立应用池的。</p> <pre><code>root@wuyfnnas:~# lsmod | grep nvme nvme 57344 0 nvme_core 184320 1 nvme t10_pi 20480 2 sd_mod,nvme_core root@wuyfnnas:~# dmesg | grep -i nvme [ 0.717077] nvme nvme0: pci function 0000:01:00.0 </code></pre> <p>哪位大神或飞牛官方人员帮看看怎么回事，谢谢！</p>

Drug

pcie

对角线

[quote][size=2][url=forum.php?mod=redirect&goto=findpost&pid=100328&ptid=21125][color=#999999]Drug 发表于 2025-4-7 11:16[/color][/url][/size] pcie[/quote]

难道我不知道nvme走的pcie通道吗 ，那叫PCI(e) Passthrough

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

你自己操作了再说吧

要么是飞牛内核不支持，要么是需要企业级m.2 nvme ssd

karuboniru

直通本身应该没问题，我之前给fnos通进去过一块PM981能识别（libvirt on Fedora 42)

建议检查下 lspci -ks 01:00.0 看看虚拟机里面的 Kernel driver in use 看看是不是nvme？

对角线

[quote][size=2][url=forum.php?mod=redirect&goto=findpost&pid=101792&ptid=21125][color=#999999]karuboniru 发表于 2025-4-9 13:31[/color][/url][/size] 直通本身应该没问题，我之前给fnos通进去过一块PM981能识别（libvirt on Fedora 42) 建议检查下 lspci -ks ...[/quote]

我上面写了啊，

root@pvenas:~# lspci -nnk -d 144d:a80d
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a801]
        Kernel driver in use: vfio-pci
        Kernel modules: nvme

144d:a80d就是01:00.0，Kernel driver in use: vfio-pci。

查资料整了1-2天，还是不行。最后用rdm直通的，虽然性能损失些，也能接受吧。磁盘控制器PCI直通企业级SATA硬盘是没问题的。

karuboniru

对角线 发表于 2025-4-10 20:28
我上面写了啊，
root@pvenas:~# lspci -nnk -d 144d:a80d
01:00.0 Non-Volatile memory controller [0108 ...

这是虚拟机里面的输出？你在虚拟机里面怎么把设备绑定给了vfio了？是这个配置错了，主机上绑定给vfio直通，但是虚拟机就让它加载默认的nvme驱动才对

对角线

[quote][size=2][url=forum.php?mod=redirect&goto=findpost&pid=104163&ptid=21125][color=#999999]karuboniru 发表于 2025-4-14 09:26[/color][/url][/size] 这是虚拟机里面的输出？你在虚拟机里面怎么把设备绑定给了vfio了？是这个配置错了，主机上绑定给vfio直通 ...[/quote]

不是的，pvenas是宿主机，是主机(宿主机)绑定的vfio的。现在已经用rdm直通了，有数据了，就不折腾nvme直通了。

还有就是我怀疑那个m.2插槽可能不是走的CPU直连，可能是南桥，所以用nvme直通一直不起作用。

21PENNY

这里软件操作很规范，没有问题，一般nvme直通用不了主要是nvme自己的问题（CFG或者BAR透传不完整）或者PCIe Partitioning，需要结合dmesg log等debug信息进行排查。