Ceph集群rbd-mirror A、B区域备份实施方案
备注：首先准备两个集群， 并确认其状态，集群的准备过程在这就不做陈述 1.查看集群状态 A区域 [root@ceph2111 ceph]# ceph -s cluster 05440e8c-bbd9-4dca-b4e7-c0c88b3fa270 health HEALTH_OK monmap e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0} election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113 osdmap e63: 4 osds: 3 up, 3 in flags sortbitwise pgmap v4121: 64 pgs, 1 pools, 557 bytes data, 14 objects 111 MB used, 45935 MB / 46046 MB avail 64 active+clean [root@ceph2111 ceph]# B区域 [root@ceph111 ceph]# ceph -s cluster e091a437-85a5-4561-b445-ef67ea98cfb2 health HEALTH_OK monmap e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0} election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115 osdmap e40: 6 osds: 6 up, 6 in flags sortbitwise pgmap v293: 64 pgs, 1 pools, 555 bytes data, 14 objects 205 MB used, 30448 MB / 30653 MB avail 64 active+clean client io 0 B/s rd, 0 op/s rd, 0 op/s wr 2.在两个集群里都安装tbd-mirror rpm包 yum install rbd-mirror 3.在A和B两端都启动rbd-mirror的进程 [root@ceph2111 ceph]# rbd-mirror -m 10.10.2.111 -d 2016-10-27 12:17:28.140639 7f788b294c40 0 ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process rbd-mirror, pid 1867 这里是用nohup 在后台运行 A区域 [root@ceph2111 ceph]# nohup rbd-mirror -m 10.10.2.111 -d > /dev/null 2>&1 & [1] 1953 B区域 [root@ceph111 ~]# nohup rbd-mirror -m 10.10.1.111 -d > /dev/null 2>&1 & [1] 15408 确认是否在运行 [root@ceph2111 ceph]# ps -ef|grep rbd root 1953 30723 0 12:19 pts/0 00:00:00 rbd-mirror -m 10.10.2.111 -d root 1984 30723 0 12:19 pts/0 00:00:00 grep --color=auto rbd 4.在ceph.conf 里添加 rbd_default_features = 125 A区域 [root@ceph2111 ceph]# vi ceph.conf [global] fsid = 05440e8c-bbd9-4dca-b4e7-c0c88b3fa270 mon_initial_members = ceph2111, ceph2112, ceph2113 mon_host = 10.10.2.111,10.10.2.112,10.10.2.113 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 2 public network = 10.10.2.0/24 rbd_default_features = 125 ~ "ceph.conf" 10L, 323C written B区域 [root@ceph111 my_cluster]# vi ceph.conf [global] fsid = e091a437-85a5-4561-b445-ef67ea98cfb2 mon_initial_members = ceph111, ceph112, ceph115 mon_host = 10.10.1.111,10.10.1.112,10.10.1.115 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 2 public network = 10.10.1.0/24 rbd_default_features = 125 5.对A进行设置 [root@ceph2111 ceph]# cp ceph.conf local.conf [root@ceph2111 ceph]# cp ceph.client.admin.keyring local.client.admin.keyring 把A区域 cp到B区域环境中 [root@ceph2111 ceph]# scp local.conf local.client.admin.keyring root@10.10.1.111:/etc/ceph/ local.conf 100% 323 0.3KB/s 00:00 local.client.admin.keyring 在A 集群里进行 同步配置 [root@ceph2111 ceph]# ceph-deploy --overwrite-conf admin ceph{2111..2114} 检验上面设置是否完成 A区域 [root@ceph2111 ceph]# ceph --cluster local mon stat e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0}, election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113 [root@ceph2111 ceph]# ceph --cluster remote mon stat e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0}, election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115 B区域 root@ceph111 ceph]# cp ceph.conf remote.conf [root@ceph111 ceph]# cp ceph.client.admin.keyring remote.client.admin.keyring [root@ceph111 ceph]# ls ceph.client.admin.keyring ceph.conf local.client.admin.keyring local.conf rbdmap remote.client.admin.keyring remote.conf tmpr_jrwC tmptBCcx2 [root@ceph111 ceph]# scp remote.c remote.client.admin.keyring remote.conf [root@ceph111 ceph]# scp remote.c* root@10.10.2.111:/etc/ceph/ remote.client.admin.keyring 100% 129 0.1KB/s 00:00 remote.conf [root@ceph111 ceph]# ceph --cluster local mon stat e5: 3 mons at {ceph2111=10.10.2.111:6789/0,ceph2112=10.10.2.112:6789/0,ceph2113=10.10.2.113:6789/0}, election epoch 44, quorum 0,1,2 ceph2111,ceph2112,ceph2113 [root@ceph111 ceph]# ceph --cluster remote mon stat e1: 3 mons at {ceph111=10.10.1.111:6789/0,ceph112=10.10.1.112:6789/0,ceph115=10.10.1.115:6789/0}, election epoch 18, quorum 0,1,2 ceph111,ceph112,ceph115 100% 320 0.3KB/s 00:00 6.准备工作完成了，开始做相关配置 A区域 [root@ceph2111 ceph]# rbd --cluster local mirror pool enable rbd pool [root@ceph2111 ceph]# rbd --cluster remote mirror pool enable rbd pool 如果需要关闭那么执行 1. rbd --cluster local mirror pool disable rbd 2. rbd --cluster remote mirror pool disable rbd 增加 CLUSTER PEER 格式 rbd mirror pool peer add {pool-name} {client-name}@{cluster-name} 使用admin这个账户就可以了 执行 A区域 [root@ceph2111 ceph]# rbd --cluster local mirror pool peer add rbd client.admin@remote d79c69c5-5fac-4bcb-8e93-28a18994d637 [root@ceph2111 ceph]# rbd --cluster remote mirror pool peer add rbd client.admin@local 9dd66d08-89e4-4c91-a389-649b9c6940de 7.查询是否连接好 A区域 [root@ceph2111 ceph]# rbd --cluster local mirror pool info Mode: pool Peers: UUID NAME CLIENT d79c69c5-5fac-4bcb-8e93-28a18994d637 remote client.admin [root@ceph2111 ceph]# rbd --cluster remote mirror pool info Mode: pool Peers: UUID NAME CLIENT 9dd66d08-89e4-4c91-a389-649b9c6940de local client.admin 如果需要删除 rbd mirror pool peer remove {pool-name} {peer-uuid} 执行 1. rbd --cluster local mirror pool peer remove image-pool uuid 2. rbd --cluster remote mirror pool peer remove image-pool uuid 8.验证是否成功 A区域 [root@ceph2111 ceph]# rbd create test2111 --size 400 开启tbd-mirror pool enable 功能 [root@ceph2111 ceph]# rbd-mirror pool enable [root@ceph2111 ceph]# rbd ls test111 test2111 [root@ceph2111 ceph]# B区域 [root@ceph111 ceph]# rbd create test111 --size 400 [root@ceph111 ceph]# rbd ls [root@ceph111 ceph]# rbd-mirror pool enable [root@ceph111 ceph]# rbd ls test111 test2111 9.备注镜像的升级与降级 1. ceph]# rbd --cluster local mirror image demote rbd/test111 2. ceph]# rbd --cluster local mirror image promote rbd/test111 3. ceph]# rbd --cluster local mirror image promote rbd/test111 4. rbd: error promoting image to primary 5. 2016-03-30 23:35:13.477096 7ffa50a3dc00 -1 librbd: image is already primary