很久没有遇到过删除Volume出错使得Volume处于Error_Deleting状态的情况了，昨天删除一个Volume时又出现了这个问题，这里顺便把解决方法记录一下。 注意我这里针对的是后端采用iscsi方式的，具体到我这里是tgt+lvm方式。
原因
目前我所遇到的删除Volume出错的原因只有一个“设备正忙”，如果你查看Volume所在的存储节点的日志就会看如下类似的内容
[-] Exception during message handling
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
rval = self.proxy.dispatch(ctxt, version, method, **args)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 206, in delete_volume
{'status': 'error_deleting'})
File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
self.gen.next()
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 195, in delete_volume
self.driver.delete_volume(volume_ref)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 203, in delete_volume
self._delete_volume(volume, volume['size'])
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 155, in _delete_volume
run_as_root=True)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 98, in _try_execute
self._execute(*command, **kwargs)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/utils.py", line 187, in execute
cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dmsetup remove -f /dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1
Exit code: 1
Stdout: ''
Stderr: 'device-mapper: remove ioctl failed: Device or resource busy
Command failed
'
那设备为什么会忙呢？首先我们在Volume所在的存储节点通过tgtadm --lld iscsi --mode target --op show 命令来查看我们要删除的Volume的信息如下
Target 7: iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 24
Initiator: iqn.1994-05.com.redhat:351f5adb9c85
Connection: 0
IP Address: 10.61.2.5
LUN information:
......
原来是我们有一个计算节点到这个Volume的链接没有释放。这里我要先说下iscsi+lvm的大致的工作方式，创建Volume时先在配置项指定的VG中创建一个LV大小Volume的大小， 然后通过tgtadm这个管理命令创建一个iscsi Target并将所创建的LV作为Target的后端存储。给实例挂载时，实例所在的计算节点调用iscsiadm命令链接到Volume对应的Target上 然后底层虚拟化软件再把这个Target指定给实例用。卸载Volume的时候就是底层虚拟化软件先从实例中移除这个Target，然后计算节点再释放它。所以如果有计算节点有链接由于某种 原因没有释放Target的话就会出现上面的情况，当时一般情况是不会出现这种问题的，但是我发现如果一个挂载有Volume的实例从一个节点迁移到另一个节点，源节点到Target的链接 并不会释放，也就是同时有两个节点链接到Target上，然后就会出现上面的问题。
解决方法
过程和原因清楚后，解决起来也就简单了，释放链接，重置Volume的状态，然后删除。首先我们登录计算节点用iscsiadm命令释放链接
[root@stack5 ~]# iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1 -u
修改数据库，重置Volume的状态
[root@stack5 ~]# mysql -h 10.61.2.12 -u cinder -p cinder -e "update volumes set status ='available' where id = 'c1374407-d1b3-407f-bbcc-7416756071c1'"
以前这样操作后就能正常删除Volume了，但这次竟然还报错了，看来得上绝招了
Clear capabilities
Removing volume: c1374407-d1b3-407f-bbcc-7416756071c1
[-] Exception during message handling
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
rval = self.proxy.dispatch(ctxt, version, method, **args)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 206, in delete_volume
{'status': 'error_deleting'})
File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
self.gen.next()
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 193, in delete_volume
self.driver.remove_export(context, volume_ref)
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 474, in remove_export
self.tgtadm.remove_iscsi_target(iscsi_target, 0, volume['id'])
File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/iscsi.py", line 168, in remove_iscsi_target
raise exception.ISCSITargetRemoveFailed(volume_id=vol_id)
ISCSITargetRemoveFailed: Failed to remove iscsi target for volume c1374407-d1b3-407f-bbcc-7416756071c1.
解决方法二
这个方法就比较暴力了，直接手动搞了。更新数据库标记Volume为已删除，在存储节点用tgtadm命令删除对应的Target，清空对应LV的数据，用lvremove命令删除LV，搞完收工。
[root@store2 ~]# mysql -h 10.61.2.12 -u cinder -p cinder -e "update volumes set deleted = 1, deleted_at = now() where id = 'c1374407-d1b3-407f-bbcc-7416756071c1'" [root@store2 ~]# tgtadm --lld iscsi --mode target --op delete --tid 7 [root@store2 ~]# dd if=/dev/zero of=/dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1 bs=1M [root@store2 ~]# lvremove /dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1