RDMA需要专门的RDMA网卡或者InfiniBand卡,若没有这些硬件设备则可以使用Soft-iWARP。
1. 下载ubuntu-16.04.6-server-amd64.iso镜像,新建一个虚拟机,搭好
2. uname -sr
Linux 4.4.0-24-generic
3. 下载内核image和headers文件
wget //kernel.ubuntu.com/~kernel-ppa/mainline/v4.8.17/linux-headers-4.8.17-040817_4.8.17-040817.201701090438_all.deb
wget //kernel.ubuntu.com/~kernel-ppa/mainline/v4.8.17/linux-headers-4.8.17-040817-generic_4.8.17-040817.201701090438_amd64.deb
wget //kernel.ubuntu.com/~kernel-ppa/mainline/v4.8.17/linux-image-4.8.17-040817-generic_4.8.17-040817.201701090438_amd64.deb
4. 安装内核、重启系统并再次查看内核版本
sudo dpkg -i linux-headers-4.8*.deb linux-image-4.8*.deb
reboot
5. apt换源
cp /etc/apt/sources.list /etc/apt/sources.list.bak
vim/etc/apt/sources.list
填入以下即可
# deb cdrom:[Ubuntu 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.1)]/ xenial main restricted
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted #Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main restricted multiverse universe #Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted multiverse universe #Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse #Added by software-properties
deb http://archive.canonical.com/ubuntu xenial partner
deb-src http://archive.canonical.com/ubuntu xenial partner
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted multiverse universe #Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-security multiverse
6. sudo apt update -y
7. sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
sudo apt-get build-dep linux
sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev
sudo apt-get install libibverbs1 libibcm1 libibcm-dev libibverbs-dev ibverbs-utils
8. vim/etc/udev/rules.d/40-ib.rules
#### /etc/udev/rules.d/40-ib.rules ####
KERNEL=="umad*",NAME="infiniband/%k"
KERNEL=="issm*",NAME="infiniband/%k"
KERNEL=="ucm*",NAME="infiniband/%k", MODE="0666"
KERNEL=="uverbs*",NAME="infiniband/%k", MODE="0666"
KERNEL=="uat",NAME="infiniband/%k", MODE="0666"
KERNEL=="ucma",NAME="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm",NAME="infiniband/%k", MODE="0666"
########
9. sudomodprobe rdma_cm
sudomodprobe ib_uverbs
sudomodprobe rdma_ucm
sudolsmod
ls/dev/infiniband/
10. 编译
sudo apt-get install libtool autoconf
git clone https://github.com/zrlio/softiwarp
11. 编译并安装userlib库
cd softiwarp/userlib/
./autogen.sh
./configure
sudo make install
sudo ln -s/usr/local/etc/libibverbs.d /etc/libibverbs.d
12. 编译并安装kernel模块
cd softiwarp/kernel/
make
modprobe rdma_cm
modprobe ib_uverbs
modprobe rdma_ucm
sudo mkdir /lib/modules/4.8.17-040817-generic/extra
sudo cp siw.ko /lib/modules/4.8.17-040817-generic/extra
insmod /lib/modules/4.8.17-040817-generic/extra/siw.ko
depmod
modprobe siw
lsmod| grep rdma
ls/dev/infiniband/
sudo cp ./userlib/src/.libs/libsiw-rdmav2.so/lib/x86_64-linux-gnu/
sudo ldconfig
modprobe siw
13. 查看IB设备和信息
另外一种在Linux机器上设置SofitWARP软件RDMA设备(这个没有进行试验):
https://github.com/animeshtrivedi/blog/blob/master/post/2019-06-26-siw.md
参考文档:
http://www.rosoo.net/a/201804/17530.html
https://weibo.com/p/2304186de3aa8a0102wpmf