Glusterfs分布式存储软件可以创建集群文件设备，但有时我们需要将文件设备转换为block设备。如下示例为Glusterfs + tcmu，将文件设备转换为block设备。
- 环境说明：
主机名
IP
设备
存储节点
oel7db01
192.168.56.201
在gluster上创建一个3副本的volume,然后在其中的一个存储节点上挂载这个volume，然后在这个gluster文件系统上创建5个文件，这5个文件后期用来模拟成block设备
oel7db02
192.168.56.202
oel7db03
192.168.56.203
计算节点
oel7db04
192.168.56.204
每个计算节点上最终会看到5个block设备，每个block设备来自于3个链路（因为有3个存储节点）
oel7db05
192.168.56.205
- 存储节点安装软件：
yum install -y glusterfs-server
yum install -y tcmu-runner
yum install -y tcmu-runner-handler-glfs
安装完成后，存储节点可以看到如下的初始状态：
[root@oel7db01 network-scripts]# targetcli targetcli shell version 2.1.fb46 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'.
/> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] | o- user:glfs .............................................................................................. [Storage Objects: 0] | o- user:qcow .............................................................................................. [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] /> |
- 存储节点配置
fallocate -l 2G file1.img fallocate -l 2G file2.img fallocate -l 2G file3.img fallocate -l 2G file4.img fallocate -l 2G file5.img
运行targetcli命令进入targetcli中，继续运行如下命令： /backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img /backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img /backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img /backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img /backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img
/iscsi/ create iqn.2016-04.org.gluster:oel7db02
从每个客户端获取iscsi的iqn: more /etc/iscsi/initiatorname.iscsi
创建本存储节点到该客户端的iqn连接 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c204 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c205
/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN1 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN2 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN3 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN4 /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN5
saveconfig
需要在所有的存储节点上运行以上步骤。 但在/backstores/user:glfs下创建LUN时，指定为当前节点的volume,例如：当前节点为oel7db02,则使用的命令如下（同时需要从第1个存储节点上获取刚刚生成的WWN号）： /backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img 965b4d0e-fedc-4729-9748-625a397c5f00 /backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img 69317210-2ea1-4c8b-87b3-98e52b35a788 /backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img 0f7da034-945b-4da6-9fa3-1c54ae6a8d66 /backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img d8304b6f-a77e-49cc-8a4e-fa95a34cbd93 /backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img f0594c4c-8e43-4a30-abf1-e0beb3129e3c |
最终存储节点配置如下所示：
4、计算节点配置
iscsiadm -m discovery -t sendtargets -p 192.168.56.201 iscsiadm -m discovery -t sendtargets -p 192.168.56.202 iscsiadm -m discovery -t sendtargets -p 192.168.56.203 iscsiadm -m node -l
最终，计算节点能看到所有的盘： [root@oel7db04 ~]# fdisk -l |grep dev Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 83886079 40893440 8e Linux LVM Disk /dev/mapper/ol_oel7db-root: 37.7 GB, 37706792960 bytes, 73646080 sectors Disk /dev/mapper/ol_oel7db-swap: 4160 MB, 4160749568 bytes, 8126464 sectors Disk /dev/sdb: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdc: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdd: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sde: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdf: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdg: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdh: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdi: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdj: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdk: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdl: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdm: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdn: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdo: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sdp: 2147 MB, 2147483648 bytes, 4194304 sectors [root@oel7db04 ~]#
安装multipath: # yum install -y device-mapper-multipath
cat > /etc/multipath.conf <<-EOF # # LIO iSCSI devices { device { vendor "LIO-ORG" path_grouping_policy "multibus" path_selector "queue-length 0" path_checker "directio" prio "const" rr_weight "uniform" } }
EOF
查看链路： [root@oel7db04 ~]# multipath -ll 36001405f0594c4c8e434a30abf1e0beb dm-6 LIO-ORG ,TCMU device size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 3:0:0:4 sdf 8:80 active ready running |- 4:0:0:4 sdk 8:160 active ready running `- 5:0:0:4 sdp 8:240 active ready running 36001405d8304b6fa77e49cc8a4efa95a dm-5 LIO-ORG ,TCMU device size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 3:0:0:3 sde 8:64 active ready running |- 4:0:0:3 sdj 8:144 active ready running `- 5:0:0:3 sdo 8:224 active ready running 360014050f7da034945b4da69fa31c54a dm-4 LIO-ORG ,TCMU device size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 3:0:0:2 sdd 8:48 active ready running |- 4:0:0:2 sdi 8:128 active ready running `- 5:0:0:2 sdn 8:208 active ready running 36001405965b4d0efedc47299748625a3 dm-2 LIO-ORG ,TCMU device size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 3:0:0:0 sdb 8:16 active ready running |- 4:0:0:0 sdg 8:96 active ready running `- 5:0:0:0 sdl 8:176 active ready running 36001405693172102ea14c8b87b398e52 dm-3 LIO-ORG ,TCMU device size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='queue-length 0' prio=1 status=active |- 3:0:0:1 sdc 8:32 active ready running |- 4:0:0:1 sdh 8:112 active ready running `- 5:0:0:1 sdm 8:192 active ready running [root@oel7db04 ~]#
[root@oel7db04 disks]# lsscsi [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0 [2:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda [3:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdb [3:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdc [3:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdd [3:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sde [3:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdf [4:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdg [4:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdh [4:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdi [4:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sdj [4:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdk [5:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdl [5:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdm [5:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdn [5:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sdo [5:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdp [root@oel7db04 disks]# |
至此，计算节点可以识别到在glusterfs上创建的5个文件设备，已经成功转换为block设备。
后期，只需要将这些block设备通过udev进行绑定即可。