基于MHA的MySQL高可用方案

MHA 简介

MHA（Master High Availability）目前在 MySQL 高可用方面是一个相对成熟的解决方案，

它由日本 DeNA 公司的 youshimaton 员工（现就职于 Facebook 公司）开发，是一套优秀的作

为 MySQL 高可用性环境下故障切换和主从角色提升的高可用软件。在 MySQL 故障切换过程

中，MHA 能做到在 0~30 秒之内自动完成数据库的主从故障切换操作，并且在进行故障切换

的过程中，MHA 能在最大程度上保证数据的一致性，以达到真正意义上的高可用。

MHA 由两部分组成：MHA Manager（管理节点）和 MHA Node（数据节点）。MHA Manager

可以单独部署在一台独立的机器上管理多个 master-slave 集群，也可以部署在一台 slave 节

点上。MHA Node 运行在每台 MySQL 服务器及 Manager 服务器上，MHA Manager 会定时探

测集群中的 master 节点，当 master 出现故障时，它可以自动将拥有最新数据的 slave 提升

为新的 master，然后将所有其他的 slave 重新指向新提升的 master。整个故障转移过程对应

用程序层面完全透明。

在 MHA 自动故障切换过程中，MHA 会试图从宕机的主服务器上保存二进制日志，最大

程度的保证数据不丢失，但这种操作是有概率性的。例如，如果主服务器硬件故障或无法通

过 ssh 访问，MHA 没法保存二进制日志，只进行故障转移从而丢失了最新的数据。使用 MySQL

5.5 的半同步复制，可以降低数据丢失的风险。MHA 可以与半同步复制结合起来。如果只有

一个 slave 已经收到了最新的二进制日志，MHA 可以将最新的二进制日志应用于其他所有的

slave 服务器上，因此可以保证所有节点的数据一致性。

目前 MHA 主要支持一主多从的架构，要搭建 MHA,要求一个复制集群中必须最少有三

台数据库服务器，一主二从，即一台充当 master，一台充当备用 master，另外一台充当从

库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，目前

淘宝 TMHA 已经支持一主一从。另外对于想快速搭建的可以参考：MHA 快速搭建

我们自己使用其实也可以使用 1 主 1 从，但是 master 主机宕机后无法切换，以及无法补全

binlog。master 的 mysqld 进程 crash 后，还是可以切换成功，以及补全 binlog 的。

工作流程

（1）从宕机崩溃的 master 上尝试保存二进制日志事件（binlog events）;

（2）识别含有最新更新的 slave 服务器；

（3）应用差异的中继日志（relay log）到其他的 slave；

（4）应用从 master 保存的二进制日志事件（binlog events）；

（5）提升一个 slave 为新的 master 服务器；

（6）将其他的 slave 连接指向新的 master 进行主从复制；

MHA 工具介绍

MHA 软件由两部分组成，Manager 工具包和 Node 工具包，具体的说明如下。

Manager 工具包主要包括以下几个工具：

➢ masterha_check_ssh 检查 MHA 的 SSH 配置状况

➢ masterha_check_repl 检查 MySQL 复制状况

➢ masterha_manger 启动 MHA

➢ masterha_check_status 检测当前 MHA 运行状态

➢ masterha_master_monitor 检测 master 是否宕机

➢ masterha_master_switch 控制故障转移（自动或者手动）

➢ masterha_conf_host 添加或删除配置的 server 信息

Node 工具包（这些工具通常由 MHA Manager 的脚本触发，无需人为操作）主要包括以下几

个工具：

➢ save_binary_logs 保存和复制 master 的二进制日志

➢ apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的

slave

➢ filter_mysqlbinlog 去除不必要的 ROLLBACK 事件（MHA 已不再使用这个工具）

➢ purge_relay_logs 清除中继日志（不会阻塞 SQL 线程）

关闭相关防火墙及相关安全机制

iptables -F

systemctl stop firewalld

setenforce 0

 

实验环境

主master            192.168.200.10

从master            192.168.200.11

slave1               192.168.200.12

slave2               192.168.200.13

manger机              192.168.200.14

配置环境

主master：hostname server01

从master：hostname server02

slave1: hsotname server03

slave2：hsotname server04

manger机：hostname server05

[root@server01 ~]# vim /etc/hosts

192.168.200.10 server01
192.168.200.11 server02
192.168.200.12 server03
192.168.200.13 server04
192.168.200.14 server05

[root@server01 ~]# scp /etc/hosts 192.168.200.11:/etc/

[root@server01 ~]# scp /etc/hosts 192.168.200.12:/etc/

[root@server01 ~]# scp /etc/hosts 192.168.200.13:/etc/

[root@server01 ~]# scp /etc/hosts 192.168.200.14:/etc/

 

一：安装MHA NODE

二：安装MHA MANGER

三：配置ssh密钥对验证

四：安装MySQL

五：搭建主从复制环境

六：配置MHA环境

七：vip配置管理

MHA NODE和MHA Manger包是需要在网站下载之后传到虚拟机上

 

 一：安装MHA NODE

######所有机器都需要安装epel源及相关的perl依赖包和MHA NODE#####

1：安装epel源

[root@server01 yum.repos.d]# rpm -ivh epel-release-latest-7.noarch.rpm

[root@server01 yum.repos.d]# yum -y clean all

[root@server01 yum.repos.d]# yum makecache

2.安装相关的perl依赖包

[root@server01 ~]# yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker

[root@server01 ~]# rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilderperl-ExtUtils-MakeMaker

3.安装MHA Node

[root@server01 ~]# tar xf mha4mysql-node-0.56.tar.gz

[root@server01 ~]# cd mha4mysql-node-0.56/

[root@server01 mha4mysql-node-0.56]# perl Makefile.PL

 

[root@server01 mha4mysql-node-0.56]# make && make install

 

[root@server01 mha4mysql-node-0.56]# ls -l /usr/local/bin/   #查看生成的文件
总用量 40
-r-xr-xr-x. 1 root root 16346 10月 23 10:37 apply_diff_relay_logs
-r-xr-xr-x. 1 root root  4807 10月 23 10:37 filter_mysqlbinlog
-r-xr-xr-x. 1 root root  7401 10月 23 10:37 purge_relay_logs
-r-xr-xr-x. 1 root root  7395 10月 23 10:37 save_binary_logs

二：安装MHA MANGER

注意：安装 MHA Manger 之前也需要安装 MHA Node

######manger只安装在manger机######

1.安装MHA Manger依赖的perl模块

[root@server05 ~]# yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes

[root@server05 ~]# yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm             #这个包需要之前下载好传到机器用yum安装

[root@server05 ~]# rpm -q  perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes perl-Config-Tiny

 

[root@server05 ~]# tar xf mha4mysql-manager-0.56.tar.gz
[root@server05 ~]# cd mha4mysql-manager-0.56/
[root@server05 mha4mysql-manager-0.56]# perl Makefile.PL

[root@server05 mha4mysql-manager-0.56]# make && make install

[root@server05 mha4mysql-manager-0.56]# ls -l /usr/local/bin/     #查看生成的文件
总用量 76
-r-xr-xr-x. 1 root root 16346 10月 23 10:44 apply_diff_relay_logs
-r-xr-xr-x. 1 root root  4807 10月 23 10:44 filter_mysqlbinlog
-r-xr-xr-x. 1 root root  1995 10月 23 10:56 masterha_check_repl
-r-xr-xr-x. 1 root root  1779 10月 23 10:56 masterha_check_ssh
-r-xr-xr-x. 1 root root  1865 10月 23 10:56 masterha_check_status
-r-xr-xr-x. 1 root root  3201 10月 23 10:56 masterha_conf_host
-r-xr-xr-x. 1 root root  2517 10月 23 10:56 masterha_manager
-r-xr-xr-x. 1 root root  2165 10月 23 10:56 masterha_master_monitor
-r-xr-xr-x. 1 root root  2373 10月 23 10:56 masterha_master_switch
-r-xr-xr-x. 1 root root  3879 10月 23 10:56 masterha_secondary_check
-r-xr-xr-x. 1 root root  1739 10月 23 10:56 masterha_stop
-r-xr-xr-x. 1 root root  7401 10月 23 10:44 purge_relay_logs
-r-xr-xr-x. 1 root root  7395 10月 23 10:44 save_binary_logs

三：配置ssh密钥对验证

所有的虚拟机都需要将自己的私钥传给别的虚拟机

manger机需要传给主master，从master，slave1，slave2，

其他虚拟机不需要传给manger

主master-->从master，slave1，slave2

从master-->主master，slave1，slave2

slave1-->主master，从master，slave2

slave2-->主master，从master，slave1

[root@server05 ~]# ssh-keygen -t rsa

[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.10

[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.11

[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.12

[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.13

四：安装MySQL

主master，从master，slave1，slave2都需要安装MySQL

[root@server01 ~]# yum -y install mariadb mariadb-server

[root@server01 ~]#systemtl start mariadb

[root@server01 ~]# mysqladmin -u root password 123456             #设置密码

五：搭建主从复制环境

1.修改MySQL的主配文件

主Master(192.168.200.10):
vim /etc/my.cnf
[mysqld]
server-id = 1
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
从Master(192.168.200.11):
vim /etc/my.cnf
[mysqld]
server-id=2
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
slave1(192.168.200.12):
vim /etc/my.cnf
[mysqld]
server-id=3
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb
slave2(192.168.200.13):
vim /etc/my.cnf
[mysqld]
server-id=4
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
systemctl restart mariadb

2.设置授权用户及主从服务器

主master

创建MySQL服务器复制授权用户

MariaDB [(none)]> grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

查看主库备份时的binlog名称和位置

MariaDB [(none)]> show master status;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-bin.000001 |      474 |              |                  |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

从master

MariaDB [(none)]> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [(none)]> change master to
    -> master_host='192.168.200.10',
    -> master_user='repl',
    -> master_password='123456',
    -> master_log_file='master-bin.000001',
    -> master_log_pos=474;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000001
          Read_Master_Log_Pos: 474
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 530
        Relay_Master_Log_File: master-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

slave1

MariaDB [(none)]> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [(none)]> change master to
    -> master_host='192.168.200.10',
    -> master_user='repl',
    -> master_password='123456',
    -> master_log_file='master-bin.000001',
    -> master_log_pos=474;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000001
          Read_Master_Log_Pos: 474
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 530
        Relay_Master_Log_File: master-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
slave2

MariaDB [(none)]> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [(none)]> change master to
    -> master_host='192.168.200.10',
    -> master_user='repl',
    -> master_password='123456',
    -> master_log_file='master-bin.000001',
    -> master_log_pos=474;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000001
          Read_Master_Log_Pos: 474
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 530
        Relay_Master_Log_File: master-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

3.设置从master，slave1，slave2的read_only状态

[root@server02 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

[root@server03 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

[root@server04 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

4.创建监控用户

需要在主master，从master，slave1，slave2做配置

MariaDB [(none)]> grant all privileges on *.* to 'root'@'192.168.200.%' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

六：配置MHA环境

只在manger机做配置

1.创建 MHA 的工作目录及相关配置文件

[root@server05 ~]# mkdir /etc/masterha
[root@server05 ~]# cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha/

2.修改app1.cnf配置文件

[root@server05 ~]# vim /etc/masterha/app1.cnf

[server default]

#设置 manager 的工作日志

manager_workdir=/var/log/masterha/app1

#设置 manager 的日志,这两条都是默认存在的

manager_log=/var/log/masterha/app1/manager.log

#设置 master 默认保存 binlog 的位置,以便 MHA 可以找到 master 日志

master_binlog_dir=/var/lib/mysql

#设置自动 failover 时候的切换脚本

master_ip_failover_script= /usr/local/bin/master_ip_failover

#设置 mysql 中 root 用户的密码

password=123456

user=root

#ping 包的时间间隔

ping_interval=1

#设置远端 mysql 在发生切换时保存 binlog 的具体位置

remote_workdir=/tmp

#设置复制用户的密码和用户名

repl_password=123456

repl_user=repl

[server1]

hostname=server01

port=3306

[server2]

hostname=server02

candidate_master=1

port=3306

check_repl_delay=0

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

3.配置故障转移脚本

[root@server05 ~]# vim /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip,

$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,

);

my $vip = '192.168.200.100'; # 写入 VIP

my $key = "1"; #非 keepalived 方式切换脚本使用的

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令

$ssh_user = "root";

GetOptions(

'command=s' => $command,

'ssh_user=s' => $ssh_user,

'orig_master_host=s' => $orig_master_host,

'orig_master_ip=s' => $orig_master_ip,

'orig_master_port=i' => $orig_master_port,

'new_master_host=s' => $new_master_host,

'new_master_ip=s' => $new_master_ip,

'new_master_port=i' => $new_master_port,

);

exit &main();

sub main {

print " IN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip=== ";

if ( $command eq "stop" || $command eq "stopssh" ) {

# $orig_master_host, $orig_master_ip, $orig_master_port are passed.

# If you manage master ip address at global catalog database,

# invalidate orig_master_ip here.

my $exit_code = 1;

#eval {

# print "Disabling the VIP on old master: $orig_master_host ";

# &stop_vip();

# $exit_code = 0;

#};

eval {

print "Disabling the VIP on old master: $orig_master_host ";

#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`;

#if ( $ping le "90.0%"&& $ping gt "0.0%" ){

#$exit_code = 0;

#}

#else {

&stop_vip();

# updating global catalog, etc

$exit_code = 0;

#}

};

if ($@) {

warn "Got Error: $@ ";

exit $exit_code;

}

exit $exit_code; 

}

elsif ( $command eq "s

tart" ) {

# all arguments are passed.

# If you manage master ip address at global catalog database,

# activate new_master_ip here.

# You can also grant write access (create user, set read_only=0, etc) here.

my $exit_code = 10;

eval {

print "Enabling the VIP - $vip on the new master - $new_master_host ";

&start_vip();

$exit_code = 0;

};

if ($@) {

warn $@;

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "status" ) {

print "Checking the Status of the script.. OK ";

`ssh $ssh_user@$orig_master_ip " $ssh_start_vip "`;

exit 0;

}

else {

&usage();

exit 1;

}

}

# A simple system call that enable the VIP on the new master

sub start_vip() {

`ssh $ssh_user@$new_master_host " $ssh_start_vip "`;

}

# A simple system call that disable the VIP on the old_master

sub stop_vip() {

`ssh $ssh_user@$orig_master_host " $ssh_stop_vip "`;

}

sub usage {

print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host

--orig_master_ip=ip --orig_master_port=port --

new_master_host=host --new_master_ip=ip --new_master_port=port "; }

[root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover                             #给脚本加执行权限

4.设置从库relay log的清除方式

从master，slave1，slave2都需要做相同的配置

[root@server02 ~]# mysql -uroot -p123456 -e 'set global relay_log_purge=0;'

[root@server03 ~]# mysql -uroot -p123456 -e 'set global relay_log_purge=0;'

[root@server04 ~]# mysql -uroot -p123456 -e 'set global relay_log_purge=0;'

 5.检查MHA ssh 通信状态

[root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Wed Oct 23 14:21:54 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Oct 23 14:21:54 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Wed Oct 23 14:21:54 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Wed Oct 23 14:21:54 2019 - [info] Starting SSH connection tests..
Wed Oct 23 14:22:06 2019 - [debug]
Wed Oct 23 14:21:57 2019 - [debug]  Connecting via SSH from root@server04(192.168.200.13:22) to root@server01(192.168.200.10:22)..
Wed Oct 23 14:21:59 2019 - [debug]   ok.
Wed Oct 23 14:21:59 2019 - [debug]  Connecting via SSH from root@server04(192.168.200.13:22) to root@server02(192.168.200.11:22)..
Wed Oct 23 14:22:01 2019 - [debug]   ok.
Wed Oct 23 14:22:01 2019 - [debug]  Connecting via SSH from root@server04(192.168.200.13:22) to root@server03(192.168.200.12:22)..
Wed Oct 23 14:22:05 2019 - [debug]   ok.
Wed Oct 23 14:22:06 2019 - [debug]
Wed Oct 23 14:21:55 2019 - [debug]  Connecting via SSH from root@server01(192.168.200.10:22) to root@server02(192.168.200.11:22)..
Wed Oct 23 14:21:57 2019 - [debug]   ok.
Wed Oct 23 14:21:57 2019 - [debug]  Connecting via SSH from root@server01(192.168.200.10:22) to root@server03(192.168.200.12:22)..
Wed Oct 23 14:22:04 2019 - [debug]   ok.
Wed Oct 23 14:22:04 2019 - [debug]  Connecting via SSH from root@server01(192.168.200.10:22) to root@server04(192.168.200.13:22)..
Wed Oct 23 14:22:05 2019 - [debug]   ok.
Wed Oct 23 14:22:08 2019 - [debug]
Wed Oct 23 14:21:56 2019 - [debug]  Connecting via SSH from root@server03(192.168.200.12:22) to root@server01(192.168.200.10:22)..
Wed Oct 23 14:22:01 2019 - [debug]   ok.
Wed Oct 23 14:22:01 2019 - [debug]  Connecting via SSH from root@server03(192.168.200.12:22) to root@server02(192.168.200.11:22)..
Wed Oct 23 14:22:06 2019 - [debug]   ok.
Wed Oct 23 14:22:06 2019 - [debug]  Connecting via SSH from root@server03(192.168.200.12:22) to root@server04(192.168.200.13:22)..
Wed Oct 23 14:22:08 2019 - [debug]   ok.
Wed Oct 23 14:22:08 2019 - [debug]
Wed Oct 23 14:21:55 2019 - [debug]  Connecting via SSH from root@server02(192.168.200.11:22) to root@server01(192.168.200.10:22)..
Wed Oct 23 14:21:58 2019 - [debug]   ok.
Wed Oct 23 14:21:58 2019 - [debug]  Connecting via SSH from root@server02(192.168.200.11:22) to root@server03(192.168.200.12:22)..
Wed Oct 23 14:22:05 2019 - [debug]   ok.
Wed Oct 23 14:22:05 2019 - [debug]  Connecting via SSH from root@server02(192.168.200.11:22) to root@server04(192.168.200.13:22)..
Wed Oct 23 14:22:07 2019 - [debug]   ok.
Wed Oct 23 14:22:08 2019 - [info] All SSH connection tests passed successfully.

6.给从服务器创建主从复制用户

从master，slave1，slave2，都要做相同的配置

MariaDB [(none)]> grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123456';
Query OK, 0 rows affected (0.39 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.11 sec)

7.设置每个主机给自己授权

主master

MariaDB [(none)]> grant all privileges on *.* to 'root'@'server01' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.11 sec)

从master

MariaDB [(none)]> grant all privileges on *.* to 'root'@'server02' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

slave1

MariaDB [(none)]> grant all privileges on *.* to 'root'@'server03' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

slave2

MariaDB [(none)]> grant all privileges on *.* to 'root'@'server04' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

8.检查整个集群的状态

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Wed Oct 23 14:41:32 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Oct 23 14:41:32 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Wed Oct 23 14:41:32 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Wed Oct 23 14:41:32 2019 - [info] MHA::MasterMonitor version 0.56.
Wed Oct 23 14:41:33 2019 - [info] Dead Servers:
Wed Oct 23 14:41:33 2019 - [info] Alive Servers:
Wed Oct 23 14:41:33 2019 - [info]   server01(192.168.200.10:3306)
Wed Oct 23 14:41:33 2019 - [info]   server02(192.168.200.11:3306)
Wed Oct 23 14:41:33 2019 - [info]   server03(192.168.200.12:3306)
Wed Oct 23 14:41:33 2019 - [info]   server04(192.168.200.13:3306)
Wed Oct 23 14:41:33 2019 - [info] Alive Slaves:
Wed Oct 23 14:41:33 2019 - [info]   server02(192.168.200.11:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 14:41:33 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 14:41:33 2019 - [info]     Primary candidate for the new Master (candidate_master is set)
Wed Oct 23 14:41:33 2019 - [info]   server03(192.168.200.12:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 14:41:33 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 14:41:33 2019 - [info]   server04(192.168.200.13:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 14:41:33 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 14:41:33 2019 - [info] Current Alive Master: server01(192.168.200.10:3306)
Wed Oct 23 14:41:33 2019 - [info] Checking slave configurations..
Wed Oct 23 14:41:33 2019 - [info]  read_only=1 is not set on slave server02(192.168.200.11:3306).
Wed Oct 23 14:41:33 2019 - [info]  read_only=1 is not set on slave server03(192.168.200.12:3306).
Wed Oct 23 14:41:33 2019 - [info]  read_only=1 is not set on slave server04(192.168.200.13:3306).
Wed Oct 23 14:41:33 2019 - [info] Checking replication filtering settings..
Wed Oct 23 14:41:33 2019 - [info]  binlog_do_db= , binlog_ignore_db=
Wed Oct 23 14:41:33 2019 - [info]  Replication filtering check ok.
Wed Oct 23 14:41:33 2019 - [info] Starting SSH connection tests..
Wed Oct 23 14:41:39 2019 - [info] All SSH connection tests passed successfully.
Wed Oct 23 14:41:39 2019 - [info] Checking MHA Node version..
Wed Oct 23 14:41:41 2019 - [info]  Version check ok.
Wed Oct 23 14:41:41 2019 - [info] Checking SSH publickey authentication settings on the current master..
Wed Oct 23 14:41:41 2019 - [info] HealthCheck: SSH to server01 is reachable.
Wed Oct 23 14:41:42 2019 - [info] Master MHA Node version is 0.56.
Wed Oct 23 14:41:42 2019 - [info] Checking recovery script configurations on the current master..
Wed Oct 23 14:41:42 2019 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000001
Wed Oct 23 14:41:42 2019 - [info]   Connecting to root@server01(server01)..
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to master-bin.000001
Wed Oct 23 14:41:42 2019 - [info] Master setting check done.
Wed Oct 23 14:41:42 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Wed Oct 23 14:41:42 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.11 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 14:41:42 2019 - [info]   Connecting to root@192.168.200.11(server02:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 14:41:43 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.12 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 14:41:43 2019 - [info]   Connecting to root@192.168.200.12(server03:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 14:41:44 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.13 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 14:41:44 2019 - [info]   Connecting to root@192.168.200.13(server04:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 14:41:45 2019 - [info] Slaves settings check done.
Wed Oct 23 14:41:45 2019 - [info]
server01 (current master)
 +--server02
 +--server03
 +--server04

Wed Oct 23 14:41:45 2019 - [info] Checking replication health on server02..
Wed Oct 23 14:41:45 2019 - [info]  ok.
Wed Oct 23 14:41:45 2019 - [info] Checking replication health on server03..
Wed Oct 23 14:41:45 2019 - [info]  ok.
Wed Oct 23 14:41:45 2019 - [info] Checking replication health on server04..
Wed Oct 23 14:41:45 2019 - [info]  ok.
Wed Oct 23 14:41:45 2019 - [info] Checking master_ip_failover_script status:
Wed Oct 23 14:41:45 2019 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.10 --orig_master_port=3306

IN SCRIPT TEST====/sbin/ifconfig ens32:1 down==/sbin/ifconfig ens32:1 192.168.200.100===

Checking the Status of the script.. OK
Wed Oct 23 14:41:46 2019 - [info]  OK.
Wed Oct 23 14:41:46 2019 - [warning] shutdown_script is not defined.
Wed Oct 23 14:41:46 2019 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

七：vip配置管理

1.开启manger服务

##查看manger状态##

[root@server05 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).

##开启manger监控##

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_con--ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 &
[1] 62367

##再次查看manger状态##

[root@server05 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:62367) is running(0:PING_OK), master:server01

##查看manger监控日志##

[root@server05 ~]# cat /var/log/masterha/app1/manager.log
Wed Oct 23 15:02:57 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Oct 23 15:02:57 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Wed Oct 23 15:02:57 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Wed Oct 23 15:02:57 2019 - [info] MHA::MasterMonitor version 0.56.
Wed Oct 23 15:02:58 2019 - [info] Dead Servers:
Wed Oct 23 15:02:58 2019 - [info] Alive Servers:
Wed Oct 23 15:02:58 2019 - [info]   server01(192.168.200.10:3306)
Wed Oct 23 15:02:58 2019 - [info]   server02(192.168.200.11:3306)
Wed Oct 23 15:02:58 2019 - [info]   server03(192.168.200.12:3306)
Wed Oct 23 15:02:58 2019 - [info]   server04(192.168.200.13:3306)
Wed Oct 23 15:02:58 2019 - [info] Alive Slaves:
Wed Oct 23 15:02:58 2019 - [info]   server02(192.168.200.11:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 15:02:58 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 15:02:58 2019 - [info]     Primary candidate for the new Master (candidate_master is set)
Wed Oct 23 15:02:58 2019 - [info]   server03(192.168.200.12:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 15:02:58 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 15:02:58 2019 - [info]   server04(192.168.200.13:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Wed Oct 23 15:02:58 2019 - [info]     Replicating from 192.168.200.10(192.168.200.10:3306)
Wed Oct 23 15:02:58 2019 - [info] Current Alive Master: server01(192.168.200.10:3306)
Wed Oct 23 15:02:58 2019 - [info] Checking slave configurations..
Wed Oct 23 15:02:58 2019 - [info]  read_only=1 is not set on slave server02(192.168.200.11:3306).
Wed Oct 23 15:02:58 2019 - [info]  read_only=1 is not set on slave server03(192.168.200.12:3306).
Wed Oct 23 15:02:58 2019 - [info]  read_only=1 is not set on slave server04(192.168.200.13:3306).
Wed Oct 23 15:02:58 2019 - [info] Checking replication filtering settings..
Wed Oct 23 15:02:58 2019 - [info]  binlog_do_db= , binlog_ignore_db=
Wed Oct 23 15:02:58 2019 - [info]  Replication filtering check ok.
Wed Oct 23 15:02:58 2019 - [info] Starting SSH connection tests..
Wed Oct 23 15:03:05 2019 - [info] All SSH connection tests passed successfully.
Wed Oct 23 15:03:05 2019 - [info] Checking MHA Node version..
Wed Oct 23 15:03:07 2019 - [info]  Version check ok.
Wed Oct 23 15:03:07 2019 - [info] Checking SSH publickey authentication settings on the current master..
Wed Oct 23 15:03:07 2019 - [info] HealthCheck: SSH to server01 is reachable.
Wed Oct 23 15:03:08 2019 - [info] Master MHA Node version is 0.56.
Wed Oct 23 15:03:08 2019 - [info] Checking recovery script configurations on the current master..
Wed Oct 23 15:03:08 2019 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000001
Wed Oct 23 15:03:08 2019 - [info]   Connecting to root@server01(server01)..
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to master-bin.000001
Wed Oct 23 15:03:08 2019 - [info] Master setting check done.
Wed Oct 23 15:03:08 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Wed Oct 23 15:03:08 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.11 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 15:03:08 2019 - [info]   Connecting to root@192.168.200.11(server02:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 15:03:09 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.12 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 15:03:09 2019 - [info]   Connecting to root@192.168.200.12(server03:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 15:03:10 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.13 --slave_port=3306 --workdir=/tmp --target_version=5.5.64-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Oct 23 15:03:10 2019 - [info]   Connecting to root@192.168.200.13(server04:22)..
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to slave-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002
    Testing mysql connection and privileges.. done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Oct 23 15:03:10 2019 - [info] Slaves settings check done.
Wed Oct 23 15:03:10 2019 - [info]
server01 (current master)
 +--server02
 +--server03
 +--server04

Wed Oct 23 15:03:10 2019 - [info] Checking master_ip_failover_script status:
Wed Oct 23 15:03:10 2019 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.10 --orig_master_port=3306

IN SCRIPT TEST====/sbin/ifconfig ens32:1 down==/sbin/ifconfig ens32:1 192.168.200.100===

Checking the Status of the script.. OK
Wed Oct 23 15:03:10 2019 - [info]  OK.
Wed Oct 23 15:03:10 2019 - [warning] shutdown_script is not defined.
Wed Oct 23 15:03:10 2019 - [info] Set master ping interval 1 seconds.
Wed Oct 23 15:03:10 2019 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Wed Oct 23 15:03:10 2019 - [info] Starting ping health check on server01(192.168.200.10:3306)..
Wed Oct 23 15:03:11 2019 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.

关闭 MHA manager 监控，忽略操作

masterha_stop --conf=/etc/masterha/app1.cnf

开启监控后查看vip

[root@server01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:95:d3:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.10/24 brd 192.168.200.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

2.模拟主库故障

##查看当前主服务ip##

MariaDB [(none)]> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000001
          Read_Master_Log_Pos: 1067
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 1123
        Relay_Master_Log_File: master-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

##暂停主master服务##

[root@server01 ~]# systemctl stop mariadb

##再次查看ip##

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:63:40:3a brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.11/24 brd 192.168.200.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

MariaDB [(none)]> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.11
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000003
          Read_Master_Log_Pos: 1798
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 530
        Relay_Master_Log_File: master-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

现在实现了MHA的故障切换及主从提升功能

它会将故障节点的信息删除

[root@server05 ~]# vim /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
user=root

[server2]
candidate_master=1
check_repl_delay=0
hostname=server02
port=3306

[server3]
hostname=server03
port=3306

[server4]
hostname=server04
port=3306

##修复主master##

开启服务

[root@server01 ~]# systemctl start mariadb

将主master变为从master的从加入到复制集群当中

只有成为从才可以切换成主

[root@server01 ~]# mysql -uroot -p123456

从master的binlog名称及位置

MariaDB [(none)]> show master status;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-bin.000004 |     1304 |              |                  |
+-------------------+----------+--------------+------------------+

MariaDB [(none)]> stop slave;

MariaDB [(none)]> change master to

master_host='192.168.200.11',

master_user='repl',

master_password='123456',                

master_log_file='master-bin.000004',       #前面需要指定，后两行报错的时候需要指定，不报错不需要指定

master_log_pos=1304;  

                  

修改监控配置文件添加server1的配置

[root@server05 ~]# vim /etc/masterha/app1.cnf

[server01]

hostname=server01

port=3306

##检查集群状态##

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
-----------------------------------忽略部分信息-----------------------------------
Thu Aug 31 22:20:30 2017 - [info] Alive Servers:
Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.10:3306)
Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.11:3306)
Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.12:3306)
Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.13:3306)
-----------------------------------忽略部分信息-----------------------------------
server02 (current master)
+--server01
+--server03
+--server04
-----------------------------------忽略部分信息-----------------------------------
MySQL Replication Health is OK.

##开启监控##

[root@server05 masterha]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 &

##关闭现在的主master##

[root@server02 ~]# systemctl stop mariadb

 

 

 

这时VIP到新的主master

这样主master就修复好了

[root@server01 ~]# ip a | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 192.168.200.10/24 brd 192.168.200.255 scope global noprefixroute ens32
    inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1
    inet6 fe80::ae9d:ac2d:3eda:3e1/64 scope link tentative noprefixroute dadfailed
    inet6 fe80::7f75:ddef:450b:883b/64 scope link noprefixroute
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

之后将从master修复好

##修复从master##

[root@server02 ~]# systemctl start mariadb             #开启服务

[root@server02 ~]# mysql -u root -p123456              #登录    

MariaDB [(none)]> stop slave;                    
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [(none)]> CHANGE MASTER TO MASTER_HOST='192.168.200.10',MASTER_USER='repl',MASTER_PASSWORD='123456';       #指定主master
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> start slave;                    
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show slave statusG 
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000007
          Read_Master_Log_Pos: 102168
               Relay_Log_File: mariadb-relay-bin.000008
                Relay_Log_Pos: 613
        Relay_Master_Log_File: master-bin.000007
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

[root@server05 ~]# vim /etc/masterha/app1.cnf                   

[server2]

hostname=server02

candidate_master=1

port=3306

check_repl_delay=0

[root@server05 masterha]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Wed Oct 23 17:13:04 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Oct 23 17:13:05 2019 - [info]   server01(192.168.200.10:3306)
Wed Oct 23 17:13:05 2019 - [info]   server02(192.168.200.11:3306)
Wed Oct 23 17:13:05 2019 - [info]   server03(192.168.200.12:3306)
Wed Oct 23 17:13:05 2019 - [info]   server04(192.168.200.13:3306)
Wed Oct 23 17:13:14 2019 - [info]   Slaves settings check done.
Wed Oct 23 17:13:14 2019 - [info]
server01 (current master)
 +--server02
 +--server03
 +--server04

Wed Oct 23 17:13:14 2019 - [info] Checking replication health on server02..
Wed Oct 23 17:13:14 2019 - [info]  ok.
Wed Oct 23 17:13:14 2019 - [info] Checking replication health on server03..
Wed Oct 23 17:13:14 2019 - [info]  ok.
Wed Oct 23 17:13:14 2019 - [info] Checking replication health on server04..
Wed Oct 23 17:13:14 2019 - [info]  ok.
Wed Oct 23 17:13:14 2019 - [info] Checking master_ip_failover_script status:
Wed Oct 23 17:13:14 2019 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.10 --orig_master_port=3306

IN SCRIPT TEST====/sbin/ifconfig ens32:1 down==/sbin/ifconfig ens32:1 192.168.200.100===

Checking the Status of the script.. OK
Wed Oct 23 17:13:14 2019 - [info]  OK.
Wed Oct 23 17:13:14 2019 - [warning] shutdown_script is not defined.
Wed Oct 23 17:13:14 2019 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.