一、什么是iscsi
iSCSI技术是一种由IBM公司研究开发的，是一个供硬件设备使用的可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。
二、需求分析
最近在做虚拟化，需要通过一个主机挂载来自两个iscsi服务器的targets，如图：
三、实现思路
问题：刚开始觉得这个应该问题不大，挂载第一台iscsi服务器的时候没有碰到问题，但是在挂载第二台iscsi时，发现怎样都无法挂载，挂载报错。
最终找到了正确的解决方法，在centos7/RHEL7上挂载时，两台iscsi服务器端的target命名需要一致，下面进行搭建演示：
系统：centos 7.2
iscsi客户端：192.168.1.156
iscsi服务端：192.168.1.157
iscsi服务端：192.168.1.158
首先要确保3台服务器的selinux和firewalld关闭
[root@iscsi-client ~]# setenforce 0 ; systemctl stop firewalld ; systemctl disable firewalld
iscsi服务端配置如下：
[root@iscsi-server1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk sr0 11:0 1 4G 0 rom
使用sdb作为target。为sdb进行分区并使用lvm进行发布出去，使用lvm的好处是后期便于扩展，在rhel7和centos7中，因为采用的是xfs文件系统，
lvm只能扩展，不能缩小，这点要牢记。
[root@iscsi-server1 ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x966a38a7. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-20971519, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): Using default value 20971519 Partition 1 of type Linux and of size 10 GiB is set Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@iscsi-server1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk └─sdb1 8:17 0 10G 0 part sr0 11:0 1 4G 0 rom
以上就为sdb做了分区，接下来将sdb1创建为独立的lvm
# 创建卷组 [root@iscsi-server1 ~]# vgcreate vg_iscsi_1_156 /dev/sdb1 Physical volume "/dev/sdb1" successfully created Volume group "vg_iscsi_1_156" successfully created # 创建逻辑分区 [root@iscsi-server1 ~]# lvcreate -l +2559 -n lv_iscsi_1_156 vg_iscsi_1_156 Logical volume "lv_iscsi_1_156" created. [root@iscsi-server1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- 17.47g swap centos -wi-ao---- 2.00g lv_iscsi_1_156 vg_iscsi_1_156 -wi-a----- 10.00g
lvm已经创建完毕，接下来创建target
# 安装target工具和服务 [root@iscsi-server1 ~]# yum install scsi-target-utils -y # 开始创建target [root@iscsi-server1 ~]# targetcli targetcli shell version 2.1.fb41 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0]
创建块设备，这里要注意，因为我们使用的lvm，因此不能将sdb添加到块设备，而是将逻辑卷添加进去
/> backstores/block create node1.disk /dev/sdb1 Cannot configure StorageObject because device /dev/sdb1 is already in use /> backstores/block create node1.disk /dev/vg_iscsi_1_156/lv_iscsi_1_156 Created block storage object node1.disk using /dev/vg_iscsi_1_156/lv_iscsi_1_156.
在创建wwn的时候，又踩了一个坑。
wwn命名必须 iqn.2017-01.com.xxx:server 注意这个月份 2017-01 这里必须要01 如果写成1就会报错。
# 错误写法 /> iscsi/ create iqn.2017-1.com.node1:server WWN not valid as: iqn, naa, eui # 正确写法 /> iscsi/ create iqn.2017-01.com.node1:server Created target iqn.2017-01.com.node1:server. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260.
# 创建acl，在创建acl这里一定要注意，如果准备在同一客户端挂载两个iscsi这里的acl命名一定要一致 /> iscsi/iqn.2017-01.com.node1:server/tpg1/acls create iqn.2017-01.com.node1:client Created Node ACL for iqn.2017-01.com.node1:client/> iscsi/iqn.2017-01.com.node1:server/tpg1/luns create /backstores/block/node1.disk Created LUN 0. Created LUN 0->0 mapping in node ACL iqn.2017-01.com.node1:client # 删除自动生成的监听 /> iscsi/iqn.2017-01.com.node1:server/tpg1/portals/ delete 0.0.0.0 3260 Deleted network portal 0.0.0.0:3260 # 创建绑定固定ip地址的监听 /> iscsi/iqn.2017-01.com.node1:server/tpg1/portals/ create 192.168.1.157 3260 Using default IP port 3260 Created network portal 192.168.1.157:3260. /> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- node1.disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (10.0GiB) write-thru activated] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2017-01.com.node1:server ...................................................................................... [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.2017-01.com.node1:client ......................................................................... [Mapped LUNs: 1] | | o- mapped_lun0 ............................................................................ [lun0 block/node1.disk (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 .......................................................... [block/node1.disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)] | o- portals .................................................................................................... [Portals: 1] | o- 192.168.1.157:3260 ............................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0] /> saveconfig Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json /> exit Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json
最后一定要执行 saveconfig 再退出
另一台iscsi服务端配置一样，targetcli如下：
/> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- node1:disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (10.0GiB) write-thru activated] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2017-01.com.node1:server ...................................................................................... [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.2017-01.com.node1:client ......................................................................... [Mapped LUNs: 1] | | o- mapped_lun0 ............................................................................ [lun0 block/node1:disk (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 .......................................................... [block/node1:disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)] | o- portals .................................................................................................... [Portals: 1] | o- 192.168.1.158:3260 ............................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0]
其中acl命名一定要和server1保持一致
这里要注意，target服务建议要配置开机启动。在搭建的时候，有出现过没有开机启动配置丢失的情况。
[root@iscsi-server2 ~]# systemctl start target ; systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
iscsi客户端配置：
[root@iscsi-client yum.repos.d]# yum install iscsi* -y
1. 修改/etc/iscsi/initiatorname.iscsi文件
[root@iscsi-client ~]# vim /etc/iscsi/initiatorname.iscsi # 这里InitiatorName 后面就是iscsi服务端通过targetcli创建的iscsi acl命名，因为要挂载两个iscsi所以名称必须一致 InitiatorName=iqn.2017-01.com.node1:client
2. 启动服务
# 这里客户端依然建议开机启动
[root@iscsi-client ~]# systemctl start iscsid ; systemctl enable iscsid
3. 挂载iscsi
# 挂载192.168.1.157，挂载之前必须要发现该iscsi [root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.157 192.168.1.157:3260,1 iqn.2017-01.com.node1:server [root@iscsi-client ~]# iscsiadm -m node -T iqn.2017-01.com.node1:server -p 192.168.1.157 -l Logging in to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.157,3260] (multiple) Login to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.157,3260] successful. # 挂载192.168.1.158，挂载之前必须要发现该iscsi [root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.158 192.168.1.158:3260,1 iqn.2017-01.com.node1:server [root@iscsi-client ~]# iscsiadm -m node -T iqn.2017-01.com.node1:server -p 192.168.1.158 -l Logging in to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.158,3260] (multiple) Login to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.158,3260] successful.
查看是否生效：
[root@iscsi-client ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk sdc 8:32 0 10G 0 disk sr0 11:0 1 4G 0 rom /mnt/iso
sdb 和 sdc都已经生成，说明挂载成功
在配置完成iscsi客户端的时候，建议进行重启服务器进行验证
# 重启成功，查看没问题。 [root@iscsi-client ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk sdc 8:32 0 10G 0 disk sr0 11:0 1 4G 0 rom
四、对iscsi进行扩容的操作
对192.168.1.157这台主机添加一块硬盘，并扩容客户端iscsi的容量
192.168.1.157 iscsi服务端操作如下：
# 已经为157添加了一块新的硬盘 sdc，首先将sdc加入到lvm中 [root@iscsi-server1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk └─sdb1 8:17 0 10G 0 part └─vg_iscsi_1_156-lv_iscsi_1_156 253:2 0 10G 0 lvm sdc 8:32 0 10G 0 disk sr0 11:0 1 4G 0 rom # 对sdc进行分区 [root@iscsi-server1 ~]# fdisk /dev/sdc Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x68a23069. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-20971519, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): Using default value 20971519 Partition 1 of type Linux and of size 10 GiB is set Command (m for help): Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@iscsi-server1 ~]# [root@iscsi-server1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk └─sdb1 8:17 0 10G 0 part └─vg_iscsi_1_156-lv_iscsi_1_156 253:2 0 10G 0 lvm sdc 8:32 0 10G 0 disk └─sdc1 8:33 0 10G 0 part sr0 11:0 1 4G 0 rom # 将sdc1添加到之前lvm卷组vg_iscsi_1_156里 [root@iscsi-server1 ~]# vgextend vg_iscsi_1_156 /dev/sdc1 Physical volume "/dev/sdc1" successfully created Volume group "vg_iscsi_1_156" successfully extended # 扩展逻辑卷 [root@iscsi-server1 ~]# lvresize -l +2559 -n /dev/vg_iscsi_1_156/lv_iscsi_1_156 Size of logical volume vg_iscsi_1_156/lv_iscsi_1_156 changed from 10.00 GiB (2559 extents) to 19.99 GiB (5118 extents). Logical volume lv_iscsi_1_156 successfully resized. # 重读下逻辑卷大小 [root@iscsi-server1 ~]# resize2fs /dev/vg_iscsi_1_156/lv_iscsi_1_156 resize2fs 1.42.9 (28-Dec-2013) resize2fs: Device or resource busy while trying to open /dev/vg_iscsi_1_156/lv_iscsi_1_156 Couldn't find valid filesystem superblock.
到此，iscsi服务端扩容操作完毕。
192.168.1.156 iscsi客户端操作如下：
重新加载sdb和sdc容量，可以看到，sdc已经变成了20G
[root@iscsi-client ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 20G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 19.5G 0 part ├─centos-root 253:0 0 17.5G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk sdc 8:32 0 20G 0 disk sr0 11:0 1 4G 0 rom
这里验证一个问题，因为挂载了两台iscsi，尝试多次重启，看看sdb和sdc会不会出现互换设备名的情况
经过3次重启的验证，并没有出现过互换设备名的情况，但是在挂载的时候，这里还是建议使用UUID
我在虚拟机上分区后无法查看到UUID，物理设备做iscsi服务端是可以查看的。
[root@db1 ~]# blkid /dev/vda1
/dev/vda1: UUID="e1b26164-3ec5-435a-bf0c-b9f4c0989941" TYPE="xfs"
至此，已经完成了一台服务器上挂载两个iscsi服务端的问题。
五、故障问题
在 192.168.1.156上查看到 sdb 所对应的是server 192.168.1.158
测试1：当sdb分区并挂载至某个目录
当158 down掉，客户端会有怎样的提示，如下验证：
dmesg 报错信息如下：
ls 挂载目录错误如下：
很明显的I/O错误。
解决方案：再次启动server 192.168.1.158, target开机自动启动，客户端需要重新挂载到目录才会恢复正常。
测试2：当sdb分区并挂载到某个目录做为web程序存放点，故障时，看web会返回什么状态码
通过apache测试，当iscsi服务器down掉，web服务器并不会马上故障。大概5分钟后报错
查看错误日志：
所以，当iscsi客户端出现IO错误时，先查看磁盘空间是否不足，再次检查下iscsi服务端网络通信是否正常。