DRBD+Keepalived高可用

DBRD 介绍

盗用一张图说明问题。

具体详细 DRBD 相关知识，请参考：https://www.cnblogs.com/hukey/p/16317752.html

KeepAlived 介绍

Keepalived是Linux下一个轻量级别的高可用解决方案。

它与HeartBeat实现类似的功能，都可以实现服务或者网络的高可用，但是又有差别，HeartBeat是一个专业的、功能完善的高可用软件，它提供HA软件所需的基本功能，比如：心跳检测、资源接管，检测集群中的服务，在集群节点转移共享IP地址的所有者等等。HeartBeat功能强大，但是部署和使用相对比较麻烦，与HeartBeat相比，Keepalived主要是通过虚拟路由冗余来实现高可用功能，虽然它没有HeartBeat功能强大，但是Keepalived部署和使用非常的简单，所有配置只需要一个配置文件即可以完成。

DRBD 高可用实践

主机信息

主机名	ip地址	操作系统	内核版本
master	192.168.199.103	Centos7.9
backup	192.168.199.104	Centos7.9

每台主机配备一块独立的磁盘，物理机为：sdb | 虚拟机为：vdb

主机初始化

两台主机操作

• 关闭selinux 和 firewalld
• 修改主机名并添加至 /etac/hosts 文件
• 校对系统时间

创建逻辑卷

两台主机操作

分区

fdisk /dev/vdb
n
回车
回车
回车
回车
w

创建逻辑卷

vgcreate vg_drbd /dev/vdb1
lvcreate -L 5G -n lv_drbd vg_drbd

注意：这里不要格式化逻辑卷

安装DRBD90

安装drbd90 程序包

两台主机操作

rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install kmod-drbd90 drbd90-utils -y

这里安装会升级内核小版本，因此安装完成后，重启下。

重启前，内核版本：3.10.0-1127.el7.x86_64
重启后，内核版本：3.10.0-1160.31.1.el7.x86_64

配置DRBD

在master节点上修改配置，然后copy到backup 节点

[root@master(192.168.199.103) ~]#cd /etc/drbd.d/
[root@master(192.168.199.103) /etc/drbd.d]#vim global_common.conf

global {
    usage-count no;

}

common {
    handlers {
        pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
        local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
    }

    startup {
    }

    options {
    }

    disk {
        on-io-error detach;
    }

    net {
        protocol C;
        cram-hmac-alg "sha1";
        shared-secret "mydrbd";
    }
}



[root@master(192.168.199.103) /etc/drbd.d]#vim drbd0.res

resource drbd0 {
    on master {
        device /dev/drbd0;
        disk /dev/vg_drbd/lv_drbd;
        address 192.168.199.103:7789;
        meta-disk internal;
    }
    on backup {
        device /dev/drbd0;
        disk /dev/vg_drbd/lv_drbd;
        address 192.168.199.104:7789;
        meta-disk internal;
    }
}

直接将 drbd0.res 、 global_common.conf 拷贝到 backup 节点

[root@master(192.168.199.103) /etc/drbd.d]#scp drbd0.res global_common.conf backup:/etc/drbd.d/

创建drbd设备

两台主机操作

drbdadm create-md drbd0

启动drbd服务

两台主机操作

# 注意，需要两台主机一起执行
systemctl start drbd

测试drbd同步

1. 设置drbd主节点
2. 格式化drbd0并挂载
3. 模拟写入文件
4. 卸载并设置主节点为 secondary
5. 将 backup节点设置为 主 drbd，并挂在查看文件是否同步

1、2、3 步骤

[root@master(192.168.199.103) ~]#drbdadm primary drbd0 --force
[root@master(192.168.199.103) ~]#drbdadm status
drbd0 role:Primary
  disk:UpToDate
  backup role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:44.05
[root@master(192.168.199.103) ~]#mk /mnt/test
mkdir: created directory ‘/mnt/test’
[root@master(192.168.199.103) ~]#mkfs.xfs /dev/drbd0
[root@master(192.168.199.103) ~]#mount /dev/drbd0 /mnt/test/
[root@master(192.168.199.103) ~]#touch /mnt/test/file{1..10}
[root@master(192.168.199.103) ~]#ls /mnt/test/
file1  file10  file2  file3  file4  file5  file6  file7  file8  file9

4 步骤

[root@master(192.168.199.103) ~]#umount /mnt/test
[root@master(192.168.199.103) ~]#drbdadm secondary drbd0
[root@master(192.168.199.103) ~]#drbdadm status
drbd0 role:Secondary
  disk:UpToDate
  backup role:Secondary
    peer-disk:UpToDate

5 步骤

[root@backup(192.168.199.104) ~]#mkdir -pv /mnt/test
mkdir: created directory ‘/mnt/test’
[root@backup(192.168.199.104) ~]#mount /dev/drbd0 /mnt/test/
[root@backup(192.168.199.104) ~]#ls /mnt/test/
file1  file10  file2  file3  file4  file5  file6  file7  file8  file9

通过上面简单测试，drbd数据同步是 ok 的。

安装keepalived

两台主机操作

yum install -y keepalived

编写配置文件及脚本

master节点操作

[root@master(192.168.199.103) /etc/keepalived]#cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_MASTER
}

vrrp_script chk_drbd {
    script "/etc/keepalived/check_drbd.sh"	### 服务检查脚本，需要自行编写
    interval 5
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
        chk_drbd
    }

    notify_stop /etc/keepalived/notify_stop.sh	### keepalived服务关闭执行的脚本
    notify_master /etc/keepalived/notify_master.sh	### 重新拿回 master 的脚本
    virtual_ipaddress {
        192.168.199.120   ### vip 请选择一个没有在占用的IP地址
    }
}

通过上面的配置文件，这里需要编写三个脚本，如下：

/etc/keepalived/check_drbd.sh

[root@master(192.168.199.103) /etc/keepalived]#cat check_drbd.sh
#!/bin/bash
# Author:hukey
systemctl status drbd
if [ $? -ne 0 ]; then
    systemctl restart drbd
    if [ $? -ne 0 ]; then
        umount /dev/drbd0
        drbdadm secondary drbd0
        systemctl stop keepalived
    fi
fi

/etc/keepalived/notify_stop.sh

[root@master(192.168.199.103) /etc/keepalived]#cat notify_stop.sh
#!/bin/bash
# Author:hukey
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_stop------\n" >> /etc/keepalived/logs/notify_stop.log
/usr/sbin/fuser -k /mnt/test &>> /etc/keepalived/logs/notify_stop.log
/bin/umount /mnt/test &>> /etc/keepalived/logs/notify_stop.log
echo -e "\n" >> /etc/keepalived/logs/notify_stop.log

/etc/keepalived/notify_master.sh

[root@master(192.168.199.103) /etc/keepalived]#cat notify_master.sh
#!/bin/bash
# Author:hukey
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_master------\n" >> /etc/keepalived/logs/notify_master.log
systemctl start drbd &>> /etc/keepalived/logs/notify_master.log
for (( i=1; i <=3; i++))
do
    /sbin/drbdadm status | egrep -q 'Primary'
    if [ $? -ne 0 ]; then
        /bin/mount /dev/drbd0 /mnt/test &>> /etc/keepalived/logs/notify_master.log
        break
    else
        sleep 3
    fi
done
echo -e "\n" >> /etc/keepalived/logs/notify_master.log

脚本所需目录和命令

[root@master(192.168.199.103) ~]#mkdir -pv /etc/keepalived/logs
mkdir: created directory ‘/etc/keepalived/logs’
[root@master(192.168.199.103) ~]#yum install -y psmisc

------------------------master节点操作完毕------------------------

backup节点操作

[root@backup(192.168.199.104) /etc/keepalived]#cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id DRBD_HA_BACKUP
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    notify_master /etc/keepalived/notify_master.sh	### master 宕机切换到backup后执行脚本
    notify_backup /etc/keepalived/notify_backup.sh	### master 节点恢复拿回主节点权限脚本
    virtual_ipaddress {
        192.168.199.120	### vip
    }
}

/etc/keepalived/notify_master.sh

[root@backup(192.168.199.104) /etc/keepalived]#cat notify_master.sh
#!/bin/bash
# Author:hukey
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_master------\n" >> /etc/keepalived/logs/notify_master.log
systemctl start drbd &>> /etc/keepalived/logs/notify_master.log
for (( i=1; i <=3; i++))
do
    /sbin/drbdadm status | egrep -q 'Primary'
    if [ $? -ne 0 ]; then
        /bin/mount /dev/drbd0 /mnt/test &>> /etc/keepalived/logs/notify_master.log
        break
    else
        sleep 3
    fi
done
echo -e "\n" >> /etc/keepalived/logs/notify_master.log

/etc/keepalived/notify_backup.sh

[root@backup(192.168.199.104) /etc/keepalived]#cat notify_backup.sh
#!/bin/bash
# Author:hukey
time=`date "+%F  %H:%M:%S"`
echo -e "$time  ------notify_backup------\n" &>> /etc/keepalived/logs/notify_backup.log
/usr/sbin/fuser -k /mnt/test &>> /etc/keepalived/logs/notify_backup.log
/bin/umount /dev/drbd0 &>> /etc/keepalived/logs/notify_backup.log
echo $time
echo -e "\n" &>> /etc/keepalived/logs/notify_backup.log

脚本所需目录和命令

[root@master(192.168.199.103) ~]#mkdir -pv /etc/keepalived/logs
mkdir: created directory ‘/etc/keepalived/logs’
[root@master(192.168.199.103) ~]#yum install -y psmisc

启动服务

两台主机操作

systemctl enable drbd keepalived
systemctl start drbd keepalived

启动后查看 master 节点 VIP 和 目录是否挂载：

[root@master(192.168.199.103) ~]#ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:eb:52:23 brd ff:ff:ff:ff:ff:ff
    inet 192.168.199.103/24 brd 192.168.199.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.120/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feeb:5223/64 scope link
       valid_lft forever preferred_lft forever

[root@master(192.168.199.103) ~]#df | egrep drbd
/dev/drbd0              xfs       5.0G   33M  5.0G   1% /mnt/test

注意：

​ 如果发现 vip 或者 drbd 没有挂载，请检查 drbd 是否存在脑裂的情况。

测试

1. 关闭 master 节点 keepalived 服务

[root@master(192.168.199.103) ~]#systemctl stop keepalived

2. 查看 backup 节点是否升级为主节点

[root@backup(192.168.199.104) ~]#ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:e9:99:cf brd ff:ff:ff:ff:ff:ff
    inet 192.168.199.104/24 brd 192.168.199.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.120/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fee9:99cf/64 scope link
       valid_lft forever preferred_lft forever
[root@backup(192.168.199.104) ~]#df | egrep drbd
/dev/drbd0              xfs       5.0G   33M  5.0G   1% /mnt/test

3. 重启 master 节点

[root@master(192.168.199.103) ~]#reboot

//启动完成后查看
[root@master(192.168.199.103) ~]#ip addr show eth0 ; df | egrep drbd
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:eb:52:23 brd ff:ff:ff:ff:ff:ff
    inet 192.168.199.103/24 brd 192.168.199.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.120/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feeb:5223/64 scope link
       valid_lft forever preferred_lft forever
/dev/drbd0              xfs       5.0G   33M  5.0G   1% /mnt/test

// master 又重新夺回了 主权