前面讲了QEMU的qcow2格式的internal snapshot和external snapshot，这都是虚拟机文件格式的功能。
这是文件级别的。
还可以是文件系统级别的，比如很多文件系统支持snapshot，如OCFS2
还可以是block级别的，比如LVM支持snapshot
我们这节来分析openstack中各种snapshot的实现。
在Openstack中，Instance的启动大概有两种，一种是从image启动，一种是从bootable volume启动
启动了的instance还可以attach一个volume。
从image启动并且attach一个volume的libvirt xml里面有
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/59ca11ea-0978-4f7d-8385-480649e63a1d/disk'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/disk/by-path/ip-16.158.166.197:3260-iscsi-iqn.2010-10.org.openstack:volume-f6ba87f7-d0b6-4fdb-ac82-346371e78c48-lun-1'/>
<target dev='vdb' bus='virtio'/>
<serial>f6ba87f7-d0b6-4fdb-ac82-346371e78c48</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
从bootable volume启动的instance的libvirt xml里面有
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/disk/by-path/ip-16.158.166.197:3260-iscsi-iqn.2010-10.org.openstack:volume-640a10f7-3965-4a47-9641-002a94526444-lun-1'/>
<target dev='vda' bus='virtio'/>
<serial>640a10f7-3965-4a47-9641-002a94526444</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
snapshot可以分为下面的几种
- 对Instance进行snapshot
- 对image启动的instance进行snapshot
- 将运行当中的instance的Ephemeral disk打成snapshot，然后上传到glance上去
- 根据虚拟机的状态和libvirt的版本，又分为live snapshot和code snapshot
- 对instance进行snapshot的时候，attached volume不会同时snapshot
- 对bootable volume启动的instance进行snapshot
- 对运行中的instance snapshot的时候，发现disk时volume，则调用cinder，对LVM进行snapshot
- snapshot的metadata会出现在glance里面，但是并不上传到glance上去，而是在LVM里面
- 这个snapshot也会出现在cinder的数据库里面
- 对volume进行snapshot：
- 最终调用cinder，对backend的LVM进行snapshot
- 这个snapshot会出现在cinder的数据库里面
所以snapshot本质上就两种，对libvirt的disk进行snapshot和对LVM进行snapshot
要进行snapshot调用的命令为
nova --debug image-create d9793e05-111c-43bb-93ed-672e94ad096e myInstanceWithVolume-snapshot3
调用的rest为
curl -i 'http://16.158.166.197:8774/v2/c24c59846a7f44538d958e7548cc74a3/servers/d9793e05-111c-43bb-93ed-672e94ad096e/action' -X POST -H "X-Auth-Project-Id: openstack" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: +Onwmvjs6T0CELsN48ON4PUNMUhF-" -d '{"createImage": {"name": "myInstanceWithVolume-snapshot3", "metadata": {}}}'
会调用/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py中的
@wsgi.response(202)
@wsgi.serializers(xml=FullServerTemplate)
@wsgi.deserializers(xml=ActionDeserializer)
@wsgi.action('createImage')
@common.check_snapshots_enabled
def _action_create_image(self, req, id, body):
在这个函数中
#如果是volume则调用volume snapshot
image = self.compute_api.snapshot_volume_backed(
context,
instance,
image_meta,
image_name,
extra_properties=props)
#如果是普通的instance则进行普通的snapshot
image = self.compute_api.snapshot(context,
instance,
image_name,
extra_properties=props)
我们先来分析volume snapshot
在/usr/lib/python2.7/dist-packages/nova/compute/api.py中有函数
@check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED])
def snapshot_volume_backed(self, context, instance, image_meta, name,
extra_properties=None):
它首先会
#调用volume的api来snapshot
snapshot = self.volume_api.create_snapshot_force(
context, volume['id'], name, volume['display_description'])
然后
#然后将这个snapshot的metadata加入glance
return self.image_service.create(context, image_meta, data='')
在/usr/lib/python2.7/dist-packages/cinder/volume/api.py中有函数
def create_snapshot_force(self, context,
volume, name,
description, metadata=None):
return self._create_snapshot(context, volume, name, description,
True, metadata)
在_create_snapshot函数中
#在cinder数据库里面创建
snapshot = self.db.snapshot_create(context, options)
#创建volume snapshot
self.volume_rpcapi.create_snapshot(context, volume, snapshot)
在/usr/lib/python2.7/dist-packages/cinder/volume/manager.py的中有函数
def create_snapshot(self, context, volume_id, snapshot_id):
#会调用driver创建snapshot
model_update = self.driver.create_snapshot(snapshot_ref)
volume_driver默认是cinder.volume.drivers.lvm.LVMISCSIDriver，它继承于LVMVolumeDriver
LVMVolumeDriver有函数
def create_snapshot(self, snapshot):
"""Creates a snapshot."""
self.vg.create_lv_snapshot(self._escape_snapshot(snapshot['name']),
snapshot['volume_name'],
self.configuration.lvm_type)
这里的vg是
self.vg = lvm.LVM(self.configuration.volume_group,
root_helper,
lvm_type=self.configuration.lvm_type,
executor=self._execute)
在/usr/lib/python2.7/dist-packages/cinder/brick/local_dev/lvm.py中有函数
def create_lv_snapshot(self, name, source_lv_name, lv_type='default'):
它主要执行了下面的命令
cmd = ['lvcreate', '--name', name,
'--snapshot', '%s/%s' % (self.vg_name, source_lv_name)]
lvcreate --size 100M --snapshot --name snap /dev/vg00/lvol1
我们再来分析instance snapshot
在/usr/lib/python2.7/dist-packages/nova/compute/api.py中有函数
@wrap_check_policy
@check_instance_cell
@check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED,
vm_states.PAUSED, vm_states.SUSPENDED])
def snapshot(self, context, instance, name, extra_properties=None):
#在glance中创建一个image的记录
image_meta = self._create_image(context, instance, name,
'snapshot',
extra_properties=extra_properties)
#真正创建snapshot
self.compute_rpcapi.snapshot_instance(context, instance,
image_meta['id'])
在/usr/lib/python2.7/dist-packages/nova/compute/manager.py中有函数
@wrap_exception()
@reverts_task_state
@wrap_instance_fault
@delete_image_on_error
def snapshot_instance(self, context, image_id, instance):
它调用_snapshot_instance，并最终调用driver
self.driver.snapshot(context, instance, image_id,
update_task_state)
nova compute的driver是libvirt
在/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py中有函数
1461 def snapshot(self, context, instance, image_href, update_task_state):
是snapshot的核心函数
1）得到libvirt的domain
virt_dom = self._lookup_by_name(instance['name'])
(Pdb) p instance['name']
'instance-0000000d'
(Pdb) p virt_dom
<libvirt.virDomain object at 0x7f22a87227d0>
2）得到image service
(image_service, image_id) = glance.get_remote_image_service(context, instance['image_ref'])
(Pdb) p image_service
<nova.image.glance.GlanceImageService object at 0x7f22a8722bd0>
(Pdb) p image_id
u'd96b0e41-8264-41de-8dbb-6b31ce9bfbfc'
3) 得到image的metadata
base = compute_utils.get_image_metadata(context, image_service, image_id, instance)
(Pdb) p base
{u'min_disk': 20, u'container_format': u'bare', u'min_ram': 0, u'disk_format': u'qcow2', 'properties': {u'instance_type_memory_mb': u'2048', u'instance_type_swap': u'0', u'instance_type_root_gb': u'20', u'instance_type_name': u'm1.small', u'instance_type_id': u'5', u'instance_type_ephemeral_gb': u'0', u'instance_type_rxtx_factor': u'1.0', u'network_allocated': u'True', u'instance_type_flavorid': u'2', u'instance_type_vcpus': u'1', u'base_image_ref': u'd96b0e41-8264-41de-8dbb-6b31ce9bfbfc'}}
4）得到snapshot
snapshot = snapshot_image_service.show(context, snapshot_image_id)
(Pdb) p snapshot
{'status': u'queued', 'name': u'myinstancewithvolume-snapshot6', 'deleted': False, 'container_format': u'bare', 'created_at': datetime.datetime(2014, 7, 3, 20, 48, 46, tzinfo=<iso8601.iso8601.Utc object at 0x7f22a873d8d0>), 'disk_format': u'qcow2', 'updated_at': datetime.datetime(2014, 7, 3, 20, 48, 46, tzinfo=<iso8601.iso8601.Utc object at 0x7f22a873d8d0>), 'id': u'2b2da7ba-9cba-4cc9-87da-a47666e62f57', 'owner': u'c24c59846a7f44538d958e7548cc74a3', 'min_ram': 0, 'checksum': None, 'min_disk': 20, 'is_public': False, 'deleted_at': None, 'properties': {u'instance_uuid': u'd9793e05-111c-43bb-93ed-672e94ad096e', u'instance_type_memory_mb': u'2048', u'user_id': u'5df0e89888364c0b80d47a7a426a9a67', u'image_type': u'snapshot', u'instance_type_id': u'5', u'instance_type_name': u'm1.small', u'instance_type_ephemeral_gb': u'0', u'instance_type_rxtx_factor': u'1.0', u'instance_type_root_gb': u'20', u'network_allocated': u'True', u'instance_type_flavorid': u'2', u'instance_type_vcpus': u'1', u'instance_type_swap': u'0', u'base_image_ref': u'd96b0e41-8264-41de-8dbb-6b31ce9bfbfc'}, 'size': 0}
5）得到Ephemeral disk的路径和格式
disk_path = libvirt_utils.find_disk(virt_dom)
source_format = libvirt_utils.get_disk_type(disk_path)
(Pdb) p disk_path
'/var/lib/nova/instances/d9793e05-111c-43bb-93ed-672e94ad096e/disk'
(Pdb) p source_format
'qcow2'
6）创建snapshot的metadata
metadata = self._create_snapshot_metadata(base, instance, image_format, snapshot['name'])
(Pdb) p metadata
{'status': 'active', 'name': u'myinstancewithvolume-snapshot6', 'container_format': u'bare', 'disk_format': 'qcow2', 'is_public': False, 'properties': {'kernel_id': u'', 'image_location': 'snapshot', 'image_state': 'available', 'ramdisk_id': u'', 'owner_id': u'c24c59846a7f44538d958e7548cc74a3'}}
7) 为snapshot生成一个文件名
snapshot_name = uuid.uuid4().hex
(Pdb) p snapshot_name
'7d9d745f5bf5482eb43886f6e69ed6e5'
8）得到snapshot文件夹
snapshot_directory = CONF.libvirt.snapshots_directory
(Pdb) p snapshot_directory
'/var/lib/nova/instances/snapshots'
9）调用live snapshot
(Pdb) p virt_dom
<libvirt.virDomain object at 0x7f22a87227d0>
(Pdb) p disk_path
'/var/lib/nova/instances/d9793e05-111c-43bb-93ed-672e94ad096e/disk'
(Pdb) p out_path
'/var/lib/nova/instances/snapshots/tmps2A_Uy/7d9d745f5bf5482eb43886f6e69ed6e5'
(Pdb) p image_format
'qcow2'
self._live_snapshot(virt_dom, disk_path, out_path, image_format)
10) _live_snapshot: 终止或者cancel当前的block operation
domain.blockJobAbort(disk_path, 0)
这是类似执行命令
virsh blockjob <domain> <path> [--abort] [--async] [--pivot] [--info] [<bandwidth>]
调用的是libvirt的virDomainBlockJobAbort
http://libvirt.org/html/libvirt-libvirt.html#virDomainBlockJobAbort
根据当前的job的不同以及flag的不同，结果不一样
如果job是VIR_DOMAIN_BLOCK_JOB_TYPE_PULL类型，cancel它需要很长时间，所以flag通常设为VIR_DOMAIN_BLOCK_JOB_ABORT_ASYNC，来异步cancel，等job真正被cancel了，会有一个event发出。
如果job是VIR_DOMAIN_BLOCK_JOB_TYPE_COPY，则马上停止当前操作，disk回到原来的状态
如果job是VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT，也马上停止，保持原状态不变
如果job是上面两种状态，而且flag设为VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT，则abort会失败，被告知正在进行copy或者commit操作。
11）_live_snapshot: 得到原disk的base disk的位置，生成新disk的位置
src_disk_size = libvirt_utils.get_disk_size(disk_path)
src_back_path = libvirt_utils.get_disk_backing_file(disk_path, basename=False)
(Pdb) p src_disk_size
21474836480
(Pdb) p src_back_path
'/var/lib/nova/instances/_base/ed39541b2c77cd7b069558570fa1dff4fda4f678'
disk_delta = out_path + '.delta'
(Pdb) p disk_delta
'/var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta'
12）_live_snapshot: 创建一个cow disk
libvirt_utils.create_cow_image(src_back_path, disk_delta, src_disk_size)
其实是执行下面的命令
qemu-img info /var/lib/nova/instances/_base/ed39541b2c77cd7b069558570fa1dff4fda4f678
qemu-img create -f qcow2 -o backing_file=/var/lib/nova/instances/_base/ed39541b2c77cd7b069558570fa1dff4fda4f678,size=21474836480 /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta
13）_live_snapshot: 将原来的disk复制到新的disk
domain.blockRebase(disk_path, disk_delta, 0,
libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY |
libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT |
libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)
(Pdb) p disk_path
'/var/lib/nova/instances/d9793e05-111c-43bb-93ed-672e94ad096e/disk'
(Pdb) p disk_delta
'/var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta'
经过这一步，我们可以看到新的disk中有了内容
root:/var/lib/nova/instances# qemu-img info d9793e05-111c-43bb-93ed-672e94ad096e/disk
image: d9793e05-111c-43bb-93ed-672e94ad096e/disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 379M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/ed39541b2c77cd7b069558570fa1dff4fda4f678
Format specific information:
compat: 1.1
lazy refcounts: false
root:/var/lib/nova/instances# qemu-img info /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta
image: /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 18G
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/ed39541b2c77cd7b069558570fa1dff4fda4f678
Format specific information:
compat: 1.1
lazy refcounts: false
这相当于执行了blockpull
例如
[root@moon ~]# virsh snapshot-list daisy --tree
snap1-daisy
|
+- snap2-daisy
|
+- snap3-daisy
snap3的base是snap2,snap2的base是snap1
我们想让snap2的base直接是snap1
[root@moon ~]# virsh blockpull --domain daisy --path /export/vmimgs/snap3-daisy.qcow2 --base /export/vmimgs/snap1-daisy.qcow2 --wait --verbose Block Pull: [100 %] Pull complete
做了这个后，发现snap3直接base是snap1了
[root@moon ~]# qemu-img info /export/vmimgs/snap3-daisy.qcow2 image: /export/vmimgs/snap3-daisy.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 145M cluster_size: 65536 backing file: /export/vmimgs/snap1-daisy.qcow2 [root@moon ~]#
14）_live_snapshot: 将新的disk进行压缩
libvirt_utils.extract_snapshot(disk_delta, 'qcow2', out_path, image_format)
调用下面的命令
qemu-img convert -f qcow2 -O qcow2 /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030.delta /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030
root:/var/lib/nova/instances# qemu-img info /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030
image: /var/lib/nova/instances/snapshots/tmpzfjdJS/7f8d11be9ff647f6b7a0a643fad1f030
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 862M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
15）将image上传到glance
在snapshot函数中
image_service.update(context, image_href, metadata, image_file)