glusterfs集群模拟单节点宕机修复

glusterfs 分布式复制集群环境

10.65.91.51  glusterfs-test01
10.65.91.52  glusterfs-test02
10.65.91.53  glusterfs-test03
10.65.91.54  glusterfs-test04
10.65.91.55  glusterfs-test05
10.65.91.56  glusterfs-test06

查看glusterfs 集群信息

# gluster  volume info 
 
Volume Name: dameizi
Type: Distributed-Replicate
Volume ID: a9d17073-43ea-417f-bd57-56946577e33d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: glusterfs-test01:/export/dameizi
Brick2: glusterfs-test02:/export/dameizi
Brick3: glusterfs-test03:/export/dameizi
Brick4: glusterfs-test04:/export/dameizi
Brick5: glusterfs-test05:/export/dameizi
Brick6: glusterfs-test06:/export/dameizi
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
features.trash: on
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

模拟故障

将 glusterfs-test01 服务器直接关机,生产环境模拟服务器意外宕机

将备用机器替换宕机的glusterfs-test01

• 准备备用机器
  备用机器 glusterfs-test08 -> 10.65.56.18
• 备用机器安装glusterfs 环境

yum install -y centos-release-gluster 
yum -y install glusterfs glusterfs-server glusterfs-fuse
systemctl enable glusterd
systemctl restart glusterd

• 备用机器创建 brick

mkdir /export/dameizi

• 修改所有 glusterfs 集群host 配置文件,新增加 10.65.56.18 glusterfs-test08

10.65.91.51  glusterfs-test01
10.65.91.52  glusterfs-test02
10.65.91.53  glusterfs-test03
10.65.91.54  glusterfs-test04
10.65.91.55  glusterfs-test05
10.65.91.56  glusterfs-test06
10.65.56.18  glusterfs-test08

• 在glusterfs 集群任意节点执行添加新节点操作

gluster peer probe glusterfs-test08 #添加新节点
gluster peer status        #查看新节点是否加入集群

• 在glusterfs 集群任意节点执行替换brick 操作

 gluster volume replace-brick dameizi  glusterfs-test01:/export/dameizi glusterfs-test08:/export/dameizi commit force

• 在glusterfs 集群任意节点执行删除宕机节点glusterfs-test01

gluster peer detach  glusterfs-test01

• 在glusterfs 集群任意节点执行查看自我修复进度

 gluster volume heal dameizi  info

• 查看新节点glusterfs-test08 数据目录,数据会自动平衡

[root@glusterfs-test08 dameizi]# du -sh *
6.8G    10.65.0.92_dev
7.8G    10.65.56.18_pro
4.0K    dameizi1
[root@glusterfs-test08 dameizi]#  du -sh *
6.8G    10.65.0.92_dev
31G     10.65.56.18_pro
4.0K    dameizi1

注意

• 客户端测试新增加目录的话必须要修改客户端host 文件与集群hosts 文件一致,否则会导致数据写入不到新节点