手动绑定驱动 + drivers_probe + rescan

Igb_uio内核驱动

Igb_uio驱动主要做的就是注册一个pci设备。但是igbuio_pci_driver对应的保存pci设备信息的id_table指针为空，这样在内核注册此pci设备时，会找不到匹配的设备，就不会调用igb_uio驱动中的探测probe函数，只会在/sys目录下创建Igb_uio相应的目录。

如何probe

在插入igb_uio.ko时是probe不到设备的，那是什么时候侦测到的呢？dpdk提供了一个Python脚本dpdk_nic_bind.py。

python tools/dpdk_nic_bind.py --bind=igb_uio eth1

运行上述命令就是将eth1网卡绑定到igb_uio模块。这时dmesg就会看到igb_uio模块的probe函数执行了，也就是意味着扫描到了匹配的pci设备。

经过分析dpdk_nic_bind.py，此脚本文件在上述情况下主要做了以下几步：

◆获取参数指定的网卡eth1的设备信息。使用lspci–Dvmmn查看。

Slot:   0000:06:00.1                                                                            

Class:  0200

Vendor: 8086

Device: 1521

SVendor:        15d9

SDevice:        1521

Rev:    01

可以查看到slot槽位信息、厂商号vendor ID、设备号device ID等信息。

◆unbind之前的igb模块。

将前面获取到的eth1对应的slot信息0000:06:00.1值写入igb的unbind文件。

echo 0000:06:00.1 >  /sys/bus/pci/drivers/igb/unbind

从内核代码分析此unbind的动作就是将igb模块信息和此pci设备Dev去关联。将dev->driver指针置为空，这个很重要。在内核处理pci设备注册的函数中，就算驱动的vendor ID和device ID与设备的都匹配上了，如果此设备的dev->driver指针不为空，也不会调用probe函数的。

◆bind新的igb_uio模块

将eth1设备的vendor和device ID信息写入igb_uio的new_id文件。

echo 0x8086 0x1521 >  /sys/bus/pci/drivers/igb_uio/new_id

内核中处理此步的函数为store_new_id，此函数中是将写入的vendor和device存入到此driver，也就是igb_uio的id_table，然后以此与PCI上的设备进行匹配，这个时候肯定会匹配成功，然后调用igb_uio模块的probe函数进行初始化动作。

echo 1 > /sys/bus/pci/rescan

echo 1 > /sys/bus/pci/devices/0000:00:1e.0/rescan

Other Network devices
=====================
0000:06:00.0 'Hi1822 Family (2*25GE) 0200' unused=hinic,igb_uio,vfio-pci

[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers
ahci      ehci-pci     hibmc-drm  hisi_sas_v3_hw  igb_uio  megaraid_sas  ohci-pci  pci-stub  uhci_hcd  xhci_hcd
ata_piix  exar_serial  hinic      hns3            ipmi_si  nvme          pcieport  serial    vfio-pci
[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/
0000:05:00.0  bind  module  new_id  remove_id  uevent  unbind
[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/module/
coresize  holders   initstate  parameters  rhelversion  srcversion  uevent
drivers   initsize  notes      refcnt      sections     taint       version
[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/module/drivers/
pci:vfio-pci
[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/
0000:05:00.0  bind  module  new_id  remove_id  uevent  unbind
[root@localhost dpdk-19.11]# cd /sys/bus/pci/drivers/vfio-pci/
[root@localhost vfio-pci]#

NVMe的驱动解绑和VFIO的绑定
查看pci信息，找到nvme：
 $ lspci -nn

00:04.0 Non-Volatile memory controller [0108]: Intel Corporation QEMU NVM Express Controller [8086:5845] (rev 02)
记住两个字段：00:04.0 和 8086:5845
00:04.0 全称是0000:00:04.0， 对应[[[[<domain>]:]<bus>]:][<device>][.[<func>]]
8086:5845 对应[<vendor>]:[<device>]
从内核驱动解绑
echo "0000:00:04.0" >/sys/bus/pci/devices/0000:00:04.0/driver/unbind
绑定到VFIO
echo 8086 5845 | tee /sys/bus/pci/drivers/vfio-pci/new_id
检查：
- lsblk 看不到nvme盘
- vfio下面多了一个设备文件：ls /dev/vfio/4
到此为止，已经完成了NVMe设备的解绑和绑定，接下来就可以开发用户态NVMe驱动来驱动NVMe盘，我们这里使用qemu的用户态driver来测试和学习。

 

# echo igb > /sys/bus/pci/devices/0000:08:00.0/driver_override
# echo 0000:08:00.0 > /sys/bus/pci/drivers_probe
# lspci -s 0000:08:00.0 -k
08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
Subsystem: Intel Corporation Device 0000
Kernel driver in use: igb
Kernel modules: igb

 echo 1 > /sys/bus/pci/rescan

# ethtool -i enp8s16 | grep bus
bus-info: 0000:08:10.0
# lspci -s 0000:08:10.0 -n
08:10.0 0200: 8086:10ca (rev 01)
# modprobe vfio
# modprobe vfio-pci
# echo 0000:08:10.0 > /sys/bus/pci/devices/0000:08:10.0/driver/unbind
# echo “8086 10ca” > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:10.0 > /sys/bus/pci/drivers/vfio-pci/bind
# lspci -s 0000:08:10.0 -k
08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
Subsystem: Intel Corporation Device 0000
Kernel driver in use: vfio-pci
Kernel modules: igbvf

[root@localhost vfio-pci]# lspci -n -v -s 0000:06:00.0
06:00.0 0200: 19e5:0200 (rev 45)
        Subsystem: 19e5:d139

[root@localhost vfio-pci]# ls /sys/bus/pci/devices/
0000:00:00.0  0000:00:12.0  0000:05:00.0  0000:74:03.0  0000:7b:00.0  0000:7d:00.3  0000:81:00.0  0000:ba:00.0
0000:00:08.0  0000:02:00.0  0000:06:00.0  0000:74:04.0  0000:7c:00.0  0000:80:00.0  0000:84:00.0  0000:ba:01.0
0000:00:0c.0  0000:03:00.0  0000:07:00.0  0000:7a:00.0  0000:7d:00.0  0000:80:08.0  0000:b4:02.0  0000:ba:02.0
0000:00:10.0  0000:04:00.0  0000:08:00.0  0000:7a:01.0  0000:7d:00.1  0000:80:0c.0  0000:b4:03.0  0000:bb:00.0
0000:00:11.0  0000:04:01.0  0000:74:02.0  0000:7a:02.0  0000:7d:00.2  0000:80:10.0  0000:b4:04.0  0000:bc:00.0
[root@localhost vfio-pci]# ls /sys/bus/pci/devices/:06:00.0/driver/
ls: cannot access /sys/bus/pci/devices/:06:00.0/driver/: No such file or directory
[root@localhost vfio-pci]# echo 0000:06:00.0 > /sys/bus/pci/devices/0000:06:00.0/driver/unbind
-bash: /sys/bus/pci/devices/0000:06:00.0/driver/unbind: No such file or directory
[root@localhost vfio-pci]# echo -n 0000:06:00.0 >  bind
-bash: echo: write error: No such device
[root@localhost vfio-pci]# pwd
/sys/bus/pci/drivers/vfio-pci
[root@localhost vfio-pci]# ls /dev/vfio/
25  vfio
[root@localhost vfio-pci]# echo -n  19e5:0200 > new_id 
-bash: echo: write error: Invalid argument
[root@localhost vfio-pci]# echo -n  "19e5:0200" > new_id 
-bash: echo: write error: Invalid argument
[root@localhost vfio-pci]# echo -n 0000:06:00.0 >  bind
-bash: echo: write error: No such device
[root@localhost vfio-pci]#

[root@localhost vfio-pci]# echo -n  "19e5 0200" > new_id 
[root@localhost vfio-pci]# echo -n 0000:06:00.0 >  bind
-bash: echo: write error: No such device
[root@localhost vfio-pci]# echo  0000:06:00.0 >  bind
-bash: echo: write error: No such device
[root@localhost vfio-pci]# ls /dev/vfio/
25  33  vfio
[root@localhost vfio-pci]# 

echo 0000:0b:00.0 >>/sys/bus/pci/devices/0000:0b:00.0/driver/unbind

注意：这条指令执行后/sys/bus/pci/devices/0000:0b:00.0/下的driver目录会消失

 =========================================

[root@localhost dpdk-19.11]# ls /dev/vfio/
25  33  vfio
[root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000:06:00.0/driver
0000:05:00.0  0000:06:00.0  bind  module  new_id  remove_id  uevent  unbind
[root@localhost dpdk-19.11]# echo 0000:06:00.0 >  /sys/bus/pci/devices/0000:06:00.0/driver/unbind 
[root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000:06:00.0/driver
ls: cannot access /sys/bus/pci/devices/0000:06:00.0/driver: No such file or directory
[root@localhost dpdk-19.11]# ls /dev/vfio/
25  vfio
[root@localhost dpdk-19.11]# echo -n  "19e5 0200" > new_id  /sys/bus/pci/drivers/vfio-pci/new_id 
[root@localhost dpdk-19.11]# echo -n  "19e5 0200" >   /sys/bus/pci/drivers/vfio-pci/new_id 
[root@localhost dpdk-19.11]# ls /dev/vfio/
25  33  vfio
[root@localhost dpdk-19.11]# echo -n  "0000:06:00.0"  /sys/bus/pci/drivers/vfio-pci/bind 
0000:06:00.0 /sys/bus/pci/drivers/vfio-pci/bind[root@localhost dpdk-19.11]# 
[root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000:06:00.0/driver
0000:05:00.0  0000:06:00.0  bind  module  new_id  remove_id  uevent  unbind
[root@localhost dpdk-19.11]# 

[root@localhost dpdk-19.11]# ./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio
0000:06:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio

[root@localhost dpdk-19.11]# ./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio
0000:06:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio

Network devices using kernel driver
===================================
0000:7d:00.0 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f0 drv=hns3 unused=hns_roce_hw_v2,igb_uio,vfio-pci *Active*
0000:7d:00.1 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f1 drv=hns3 unused=igb_uio,vfio-pci 
0000:7d:00.2 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f2 drv=hns3 unused=hns_roce_hw_v2,igb_uio,vfio-pci 
0000:7d:00.3 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f3 drv=hns3 unused=igb_uio,vfio-pci 

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

[root@localhost dpdk-19.11]# ls /sys/class/net/     ----没dpdk的网卡
enp125s0f0  enp125s0f1  enp125s0f2  enp125s0f3  lo

[root@localhost dpdk-19.11]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp125s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b0:08:75:5f:b7:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.10.16.229/24 brd 10.10.16.255 scope global noprefixroute enp125s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::f9ec:a960:7518:b745/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp125s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b0:08:75:5f:b7:da brd ff:ff:ff:ff:ff:ff
4: enp125s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b0:08:75:5f:b7:db brd ff:ff:ff:ff:ff:ff
5: enp125s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b0:08:75:5f:b7:dc brd ff:ff:ff:ff:ff:ff
[root@localhost dpdk-19.11]# lspci | grep ether
[root@localhost dpdk-19.11]# lspci | grep -i  ether
05:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (2*25GE) (rev 45)
06:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (2*25GE) (rev 45)
7d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21)
7d:00.1 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21)
7d:00.2 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21)
7d:00.3 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21)
[root@localhost dpdk-19.11]#