基于 MHA 的 MySQL 高可用方案

基于 MHA 的 MySQL 高可用方案

一、MHA 简介

MHA（Master High Availability）目前在 MySQL 高可用方面是一个相对成熟的解决方案， 它由日本 DeNA 公司的 youshimaton 员工（现就职于 Facebook 公司）开发，是一套优秀的作为 MySQL 高可用性环境下故障切换和主从角色提升的高可用软件。在 MySQL 故障切换过程中，MHA 能做到在 0~30 秒之内自动完成数据库的主从故障切换操作，并且在进行故障切换的过程中，MHA 能在最大程度上保证数据的一致性，以达到真正意义上的高可用。

MHA 由两部分组成：MHA Manage（r  管理节点）和 MHA Node（数据节点）。MHA Manager

可以单独部署在一台独立的机器上管理多个 master-slave 集群，也可以部署在一台 slave 节点上。MHA Node 运行在每台MySQL 服务器及Manager 服务器上，MHA Manager 会定时探测集群中的 master 节点，当 master 出现故障时，它可以自动将拥有最新数据的 slave 提升为新的 master，然后将所有其他的 slave 重新指向新提升的 master。整个故障转移过程对应用程序层面完全透明。

在 MHA 自动故障切换过程中，MHA 会试图从宕机的主服务器上保存二进制日志，最大程度的保证数据不丢失，但这种操作是有概率性的。例如，如果主服务器硬件故障或无法通过 ssh 访问，MHA 没法保存二进制日志，只进行故障转移从而丢失了最新的数据。使用 MySQL

5.5 的半同步复制，可以降低数据丢失的风险。MHA 可以与半同步复制结合起来。如果只有一个 slave 已经收到了最新的二进制日志，MHA 可以将最新的二进制日志应用于其他所有的slave 服务器上，因此可以保证所有节点的数据一致性。

目前 MHA 主要支持一主多从的架构，要搭建 MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二从，即一台充当 master，一台充当备用 master，另外一台充当从库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，目前淘宝 TMHA 已经支持一主一从。另外对于想快速搭建的可以参考：MHA 快速搭建

我们自己使用其实也可以使用 1 主 1 从，但是 master 主机宕机后无法切换，以及无法补全

binlog。master 的 mysqld 进程crash 后，还是可以切换成功，以及补全 binlog 的。官方介绍：https://code.google.com/p/mysql-master-ha/

1.1 、工作流程

 

（1） 从宕机崩溃的 master 上尝试保存二进制日志事件（binlog events）;

（2） 识别含有最新更新的 slave 服务器；

（3） 应用差异的中继日志（relay log）到其他的 slave；

（4） 应用从 master 保存的二进制日志事件（binlog events）；

（5） 提升一个 slave 为新的 master 服务器；

（6） 将其他的 slave 连接指向新的 master 进行主从复制；

1.2 、MHA 工具介绍

 

MHA 软件由两部分组成，Manager 工具包和 Node 工具包，具体的说明如下：

masterha_check_ssh 检查 MHA 的 SSH 配置状况Manager 工具包主要包括以下几个工具：

• masterha_check_repl 检查 MySQL 复制状况
• masterha_manger 启动 MHA
• masterha_check_status 检测当前MHA 运行状态
• masterha_master_monitor 检测 master 是否宕机
• masterha_master_switch 控制故障转移（自动或者手动）
• masterha_conf_host 添加或删除配置的 server 信息

Node 工具包（这些工具通常由 MHA Manager 的脚本触发，无需人为操作）主要包括以下几个工具：

➢ save_binary_logs 保存和复制 master 的二进制日志

➢ apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的

slave

➢ filter_mysqlbinlog 去除不必要的 ROLLBACK 事件（MHA 已不再使用这个工具）

➢ purge_relay_logs 清除中继日志（不会阻塞 SQL 线程）

注意：为了尽可能的减少主库硬件损坏宕机造成的数据丢失，因此在配置 MHA 的同时建议配置成 MySQL 5.5 的半同步复制。关于半同步复制原理各位自己进行查阅。（不是必须）

1.1 、MHA 环境说明

角色                    IP 地址                      主机名                    ServerID 数据库类型

Primary Master 192.168.200.111          server01                  1            写入

Secondary Master 192.168.200.112     server02                  2            写入

Slave1 192.168.200.113                       server03                   3            读

Slave2 192.168.200.114                       server04                   4            读

Manager 192.168.200.115                    server05 - 监控复制组

LVS Matser 192.168.200.116                localhost -

LVS Backup 192.168.200.117                 localhost -

 

二、前期环境部署

 

2.1、配置所有主机名称

 

master1 主机:

hostname server01

bash

 

master2 主机:

hostname server02

bash

 

slave1 主机:

hostname server03

bash

 

slave2 主机:

hostname server04

bash

 

manager 主机:

hostname server05

bash

 

2.2、配置所有主机名映射

 

[root@server05 ~]# vim /etc/hosts

192.168.200.111 server01

192.168.200.112 server02

192.168.200.113 server03

192.168.200.114 server04

192.168.200.115 server05

 

scp /etc/hosts 192.168.200.111:/etc/

scp /etc/hosts 192.168.200.112:/etc/

scp /etc/hosts 192.168.200.113:/etc/

scp /etc/hosts 192.168.200.114:/etc/

 

2.3、所有主机关闭防火墙和安全机制

 

systemctl stop iptables

systemctl stop firewalld

setenforce 0

 

2.4、下载 mha-manager 和 mha-node

http://downloads.mariadb.com/MHA/

 

三、安装 MHA node

 

3.1、所有主机安装 MHA node 及相关 perl 依赖包

安装 epel 源:

rpm -ivh epel-release-latest-7.noarch.rpm

 

yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder

perl-ExtUtils-MakeMaker

 

rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder

perl-ExtUtils-MakeMaker

 

perl-DBD-MySQL-4.023-6.el7.x86_64

perl-DBI-1.627-4.el7.x86_64

perl-CPAN-1.9800-292.el7.noarch

perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch

perl-ExtUtils-MakeMaker-6.68-3.el7.noarch

注意：安装后建议检查一下所需软件包是否全部安装

 

3.2、所有主机上安装 MHA Node

 

tar xf mha4mysql-node-0.56.tar.gz

cd mha4mysql-node-0.56/

perl Makefile.PL

make && make install

 

3.3、MHA Node 安装完后会在 /usr/local/bin 生成以下脚本

 

ls -l /usr/local/bin/

总用量 40

-r-xr-xr-x 1 root root 16346 12 月 29 15:07 apply_diff_relay_logs

-r-xr-xr-x 1 root root 4807 12 月 29 15:07 filter_mysqlbinlog

-r-xr-xr-x 1 root root 7401 12 月 29 15:07 purge_relay_logs

-r-xr-xr-x 1 root root 7395 12 月 29 15:07 save_binary_logs

 

四、安装 MHA Manger

注意：安装 MHA Manger 之前也需要安装 MHA Node

4.1、首先安装 MHA Manger 依赖的 perl 模块（我这里使用 yum 安装）

 

yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI

perl-Time-HiRes

 

yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm

 

rpm -q perl cpan perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI

perl-Time-HiRes perl-Config-Tiny

 

perl-5.16.3-292.el7.x86_64

perl-Log-Dispatch-2.41-1.el7.1.noarch

perl-Parallel-ForkManager-1.18-2.el7.noarch

perl-DBD-MySQL-4.023-6.el7.x86_64

perl-DBI-1.627-4.el7.x86_64

perl-Time-HiRes-1.9725-3.el7.x86_64

perl-Config-Tiny-2.14-7.el7.noarch

 

注意：之前时候 perl-Config-Tiny.noarch 没有安装成功,后来用 cpan(cpan install Config::Tiny )

 

4.2、安装 MHA Manger 软件包

 

tar xf mha4mysql-manager-0.56.tar.gz

cd mha4mysql-manager-0.56/

perl Makefile.PL

make && make install

 

4.3、安装完成后会有以下脚本文件

 

ls -l /usr/local/bin/

总用量 76

-r-xr-xr-x 1 root root 16346 12 月 29 15:08 apply_diff_relay_logs

-r-xr-xr-x 1 root root 4807 12 月 29 15:08 filter_mysqlbinlog

-r-xr-xr-x 1 root root 1995 12 月 29 15:37 masterha_check_repl

-r-xr-xr-x 1 root root 1779 12 月 29 15:37 masterha_check_ssh

-r-xr-xr-x 1 root root 1865 12 月 29 15:37 masterha_check_status

-r-xr-xr-x 1 root root 3201 12 月 29 15:37 masterha_conf_host

-r-xr-xr-x 1 root root 2517 12 月 29 15:37 masterha_manager

-r-xr-xr-x 1 root root 2165 12 月 29 15:37 masterha_master_monitor

-r-xr-xr-x 1 root root 2373 12 月 29 15:37 masterha_master_switch

-r-xr-xr-x 1 root root 3879 12 月 29 15:37 masterha_secondary_check

-r-xr-xr-x 1 root root 1739 12 月 29 15:37 masterha_stop

-r-xr-xr-x 1 root root 7401 12 月 29 15:08 purge_relay_logs

-r-xr-xr-x 1 root root 7395 12 月 29 15:08 save_binary_logs

 

五、配置 SSH 密钥对验证

 

服务器之间需要实现密钥对验证。关于配置密钥对验证可看下面步骤。但是有一点需要

注意：不能禁止 password 登陆，否则会出现错误

1. 服务器先生成一个密钥对

2. 把自己的公钥传给对方

 

5.1、Server05(192.168.200.115)上:

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

 

注意：Server05 需要连接每个主机测试，因为第一次连接的时候需要输入 yes，影响后期故

障切换时，对于每个主机的 SSH 控制。

ssh server01

ssh server02

ssh server03

ssh server04

 

5.2、Primary Master(192.168.200.111):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

 

5.3、Secondary Master(192.168.200.112):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

 

5.4、slave1(192.168.200.113):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

 

5.5、slave2(192.168.200.114):

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112

ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

 

六、安装 mysql

111-114 主机上的操作：

 

yum -y install mariadb mariadb-server mariadb-devel

systemctl start mariadb

 

netstat -lnpt | grep :3306

 

设置数据库初始密码（后续操作中使用）

 

mysqladmin -u root password 123456

 

七、搭建主从复制环境

注意：binlog-do-db 和 replicate-ignore-db 设置必须相同。 MHA 在启动时候会检测过滤规

则，如果过滤规则不同，MHA 将不启动监控和故障转移功能。

 

7.1、修改 mysql 主机的配置文件

Primary Master(192.168.200.111):

 

vim /etc/my.cnf

[mysqld]

server-id = 1

log-bin=master-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

Secondary Master(192.168.200.112):

vim /etc/my.cnf

[mysqld]

server-id=2

log-bin=master-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

slave1(192.168.200.113):

vim /etc/my.cnf

[mysqld]

server-id=3

log-bin=mysql-bin

relay-log=slave-relay-bin

log-slave-updates=true

relay_log_purge=0

systemctl restart mariadb

slave2(192.168.200.114):

vim /etc/my.cnf

[mysqld]

server-id=4

log-bin=mysql-bin

relay-log=slave-relay-bin

log-slave-updates=true

relay_log_purge=0

 

systemctl restart mariadb

 

7.2、在 Primary Master(192.168.200.111)上对旧数据进行备份 新机器不用作

mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql

参数解释：

--master-data=2 备份时刻记录 master 的 Binlog 位置和 Position

--single-transaction 获取一致性快照

-R 备份存储过程和函数

-triggres 备份触发器

-A 备份所有的库。

 

7.3、mysql 服务器创建复制授权用户

grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123456';

 

flush privileges;

 

7.4、查看主库备份时的 binlog 名称和位置

MariaDB [(none)]> show master status;

+-------------------+----------+--------------+------------------+

| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |

+-------------------+----------+--------------+------------------+

| master-bin.000001 | 474 | | |

+-------------------+----------+--------------+------------------+

1 row in set (0.00 sec)

7.5、把数据备份复制其他主机

scp all.sql 192.168.200.112:/tmp/

scp all.sql 192.168.200.113:/tmp/

scp all.sql 192.168.200.114:/tmp/

7.6、导入备份数据到 112-114 主机上执行复制相关命令

 

mysql -uroot -p123456< /tmp/all.sql

 

stop slave;

 

CHANGE MASTER TO

MASTER_HOST='192.168.200.111',

MASTER_USER='repl',

MASTER_PASSWORD='123456',

MASTER_LOG_FILE='master-bin.000001',

MASTER_LOG_POS=474;

 

start slave;

 

show slave statusG

 

 

# 检查 IO 和 SQL 线程是否为：yes

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

7.7、主从同步故障处理

Slave_IO_Running: No

Slave_SQL_Running: Yes

-----------------------------------忽略部分信息-----------------------------------

Last_IO_Errno: 1236

Last_IO_Error: Got fatal error 1236 from master when reading data from

binary log: 'Could not find first log file name in binary log index file'

-----------------------------------忽略部分信息-----------------------------------

处理方式：

stop slave;

reset slave;

set global sql_slave_skip_counter =1 ;

start slave;

7.8、三台 slave 服务器设置 read_only 状态

从库对外只提供读服务，只所以没有写进 mysql 配置文件，是因为随时 server02 会提升为

master

[root@server02 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

[root@server03 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

[root@server04 ~]# mysql -uroot -p123456 -e 'set global read_only=1'

 

7.9、创建监控用户（111-114 主机上的操作）：

 

grant all privileges on *.* to 'root'@'192.168.200.%' identified by '123456';

flush privileges;

 

为自己的主机名授权：

 

grant all privileges on *.* to 'root'@'server04' identified by '123456';

flush privileges;

 

到这里整个 mysql 主从集群环境已经搭建完毕。

八、配置 MHA 环境

8.1、创建 MHA 的工作目录及相关配置文件

Server05(192.168.200.115)：在软件包解压后的目录里面有样例配置文件

 

mkdir /etc/masterha

cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha

 

8.2、修改 app1.cnf 配置文件

 

/usr/local/bin/master_ip_failover 脚本需要根据自己环境修改 ip 和网卡名称等。

 

vim /etc/masterha/app1.cnf

 

[server default]

#设置 manager 的工作日志

manager_workdir=/var/log/masterha/app1

#设置 manager 的日志,这两条都是默认存在的

manager_log=/var/log/masterha/app1/manager.log

#设置 master 默认保存 binlog 的位置,以便 MHA 可以找到 master 日志

master_binlog_dir=/var/lib/mysql

#设置自动 failover 时候的切换脚本

master_ip_failover_script= /usr/local/bin/master_ip_failover

#设置 mysql 中 root 用户的密码

password=123456

user=root

#ping 包的时间间隔

ping_interval=1

#设置远端 mysql 在发生切换时保存 binlog 的具体位置

remote_workdir=/tmp

#设置复制用户的密码和用户名

repl_password=123456

repl_user=repl

[server1]

hostname=server01

port=3306

[server2]

hostname=server02

candidate_master=1

port=3306

check_repl_delay=0

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

 

 

8.3、配置故障转移脚本

 

[root@server05 ~]# vim /usr/local/bin/master_ip_failover

 

#!/usr/bin/env perl

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip,

$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,

);

my $vip = '192.168.200.100'; # 写入 VIP

my $key = "1"; #非 keepalived 方式切换脚本使用的

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令

$ssh_user = "root";

GetOptions(

'command=s' => $command,

'ssh_user=s' => $ssh_user,

'orig_master_host=s' => $orig_master_host,

'orig_master_ip=s' => $orig_master_ip,

'orig_master_port=i' => $orig_master_port,

'new_master_host=s' => $new_master_host,

'new_master_ip=s' => $new_master_ip,

'new_master_port=i' => $new_master_port,

);

exit &main();

sub main {

print " IN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip=== ";

if ( $command eq "stop" || $command eq "stopssh" ) {

# $orig_master_host, $orig_master_ip, $orig_master_port are passed.

# If you manage master ip address at global catalog database,

# invalidate orig_master_ip here.

my $exit_code = 1;

#eval {

# print "Disabling the VIP on old master: $orig_master_host ";

# &stop_vip();

# $exit_code = 0;

#};

eval {

print "Disabling the VIP on old master: $orig_master_host ";

#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`;

#if ( $ping le "90.0%"&& $ping gt "0.0%" ){

#$exit_code = 0;

#}

#else {

&stop_vip();

# updating global catalog, etc

$exit_code = 0;

#}

};

if ($@) {

warn "Got Error: $@ ";

exit $exit_code;

}

exit $exit_code; 

}

elsif ( $command eq "start" ) {

# all arguments are passed.

# If you manage master ip address at global catalog database,

# activate new_master_ip here.

# You can also grant write access (create user, set read_only=0, etc) here.

my $exit_code = 10;

eval {

print "Enabling the VIP - $vip on the new master - $new_master_host ";

&start_vip();

$exit_code = 0;

};

if ($@) {

warn $@;

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "status" ) {

print "Checking the Status of the script.. OK ";

`ssh $ssh_user@$orig_master_ip " $ssh_start_vip "`;

exit 0;

}

else {

&usage();

exit 1;

}

}

# A simple system call that enable the VIP on the new master

sub start_vip() {

`ssh $ssh_user@$new_master_host " $ssh_start_vip "`;

}

# A simple system call that disable the VIP on the old_master

sub stop_vip() {

`ssh $ssh_user@$orig_master_host " $ssh_stop_vip "`;

}

sub usage {

print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host

--orig_master_ip=ip --orig_master_port=port --

new_master_host=host --new_master_ip=ip --new_master_port=port "; }

 

[root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover

 

 

 

8.4、设置从库 relay log 的清除方式(112-114)：

mysql -uroot -p123456 -e 'set global relay_log_purge=0;'

注意：

MHA 在故障切换的过程中，从库的恢复过程依赖于 relay log 的相关信息，所以这里要

将 relay log 的自动清除设置为 OFF，采用手动清除 relay log 的方式。在默认情况下，从服务

器上的中继日志会在 SQL 线程执行完毕后被自动删除。但是在 MHA 环境中，这些中继日志

在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动清除功能。定期清除中

继日志需要考虑到复制延时的问题。在 ext3 的文件系统下，删除大的文件需要一定的时间，

会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在 linux

系统中通过硬链接删除大文件速度会很快。（在 mysql 数据库中，删除大表时，通常也采用

建立硬链接的方式）

8.5、配置从库（112-114）relay_log 清除脚本加入计划任务

MHA 节点中包含了 pure_relay_logs 命令工具，它可以为中继日志创建硬链接，执行 SET

GLOBAL relay_log_purge=1,等待几秒钟以便 SQL 线程切换到新的中继日志，再执行 SET

GLOBAL relay_log_purge=0。

vim purge_relay_log.sh

#!/bin/bash

user=root

passwd=123456 #注意：数据库要有密码,填自己所设置的密码就可以，前面设置过

port=3306

log_dir='/tmp'

work_dir='/tmp'

purge='/usr/local/bin/purge_relay_logs'

if [ ! -d $log_dir ]

then

mkdir $log_dir -p

fi

$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port

--workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1

chmod +x purge_relay_log.sh

crontab -e

0 4 * * * /bin/bash /root/purge_relay_log.sh

pure_relay_logs 脚本参数如下所示：

--user mysql 用户名

--password mysql 密码

--port 端口号

--workdir 指定创建 relay log 的硬链接的位置，默认是/var/tmp，由于系统不同

分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继

日志文件被删除

--disable_relay_log_purge 默认情况下，如果 relay_log_purge=1，脚本会什么都不清理，自

动退出，通过设定这个参数，当 relay_log_purge=1 的情况下会将 relay_log_purge 设置为 0。

清理 relay log 之后，最后将参数设置为 OFF。

8.6、手动清除中继日志

purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306

--workdir=/tmp

2017-08-31 21:33:52: purge_relay_logs script started.

Found relay_log.info: /usr/local/mysql/data/relay-log.info

Removing hard linked relay log files slave-relay-bin* under /tmp.. done.

Current relay log file: /usr/local/mysql/data/slave-relay-bin.000002

Archiving unused relay log files (up to /usr/local/mysql/data/slave-relay-bin.000001) ...

Creating hard link for /usr/local/mysql/data/slave-relay-bin.000001 under

/tmp/slave-relay-bin.000001 .. ok.

Creating hard links for unused relay log files completed.

Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL

thread can delete older relay log files (if i

t keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files

slave-relay-bin* under /tmp.. done.

2017-08-31 21:33:56: All relay log purging operations succeeded.

 

 

 

 

8.7、检查 MHA ssh 通信状态

[root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf

 

Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not

found. Skipping.

Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from

/etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.

最后会返回 successfully 表示没有问题

 

8.9、检查整个集群的状态

 

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not

found. Skipping.

Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from

/etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from

root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:03:59 2018 - [debug] ok.

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [debug]

Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22)..

Sat Dec 29 16:04:00 2018 - [debug] ok.

Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22)..

Sat Dec 29 16:04:01 2018 - [debug] ok.

Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from

root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..

Sat Dec 29 16:04:02 2018 - [debug] ok.

Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Sat Dec 29 16:04:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not

found. Skipping.

Sat Dec 29 16:04:53 2018 - [info] Reading application default configurations from

/etc/masterha/app1.cnf..

Sat Dec 29 16:04:53 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:04:53 2018 - [info] MHA::MasterMonitor version 0.56.

Creating directory /var/log/masterha/app1.. done.

Sat Dec 29 16:04:55 2018 - [info] Dead Servers:

Sat Dec 29 16:04:55 2018 - [info] Alive Servers:

Sat Dec 29 16:04:55 2018 - [info] server01(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306)

Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306)

Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306)

Sat Dec 29 16:04:55 2018 - [info] Alive Slaves:

Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Primary candidate for the new Master

(candidate_master is set)

Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:04:55 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)

Sat Dec 29 16:04:55 2018 - [info] Checking slave configurations..

Sat Dec 29 16:04:55 2018 - [info] Checking replication filtering settings..

Sat Dec 29 16:04:55 2018 - [info] binlog_do_db= , binlog_ignore_db=

Sat Dec 29 16:04:55 2018 - [info] Replication filtering check ok.

Sat Dec 29 16:04:55 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:05:00 2018 - [info] All SSH connection tests passed successfully.

Sat Dec 29 16:05:00 2018 - [info] Checking MHA Node version..

Sat Dec 29 16:05:02 2018 - [info] Version check ok.

Sat Dec 29 16:05:02 2018 - [info] Checking SSH publickey authentication settings on the current

master..

Sat Dec 29 16:05:03 2018 - [info] HealthCheck: SSH to server01 is reachable.

Sat Dec 29 16:05:04 2018 - [info] Master MHA Node version is 0.56.

Sat Dec 29 16:05:04 2018 - [info] Checking recovery script configurations on the current

master..

Sat Dec 29 16:05:04 2018 - [info] Executing command: save_binary_logs --command=test

--start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --

manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:05:04 2018 - [info]

Connecting to root@server01(server01)..

Creating /tmp if not exists.. ok.

Checking output directory is accessible or not..

ok.

Binlog found at /var/lib/mysql, up to master-bin.000001

Sat Dec 29 16:05:04 2018 - [info] Master setting check done.

Sat Dec 29 16:05:04 2018 - [info] Checking SSH publickey authentication and checking recovery

script configurations on all alive slave servers..

Sat Dec 29 16:05:04 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:05:04 2018 - [info] Connecting to

root@192.168.200.112(server02:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002

Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:05:05 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:05:05 2018 - [info] Connecting to

root@192.168.200.113(server03:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to slave-relay-bin.000002

Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:05:07 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:05:07 2018 - [info] Connecting to

root@192.168.200.114(server04:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to slave-relay-bin.000002

Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:05:08 2018 - [info] Slaves settings check done.

Sat Dec 29 16:05:09 2018 - [info]

server01 (current master)

+--server02

+--server03

+--server04

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server02..

Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server03..

Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server04..

Sat Dec 29 16:05:09 2018 - [info] ok.

Sat Dec 29 16:05:09 2018 - [info] Checking master_ip_failover_script status:

Sat Dec 29 16:05:09 2018 - [info] /usr/local/bin/master_ip_failover --command=status

--ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m

aster_port=3306

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Checking the Status of the script.. OK

bash: /etc/init.d/keepalived: 没有那个文件或目录

Sat Dec 29 16:05:09 2018 - [info] OK.

Sat Dec 29 16:05:09 2018 - [warning] shutdown_script is not defined.

Sat Dec 29 16:05:09 2018 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK. 

 

九、VIP 配置管理

 

Master vip 配置有两种方式,一种是通过 keepalived 或者 heartbeat 类似的软件的方式管

理 VIP 的浮动,另一种为通过命令方式管理。

通过命令方式管理 VIP 地址：

打开在前面编辑过的文件/etc/masterha/app1.cnf，检查如下行是否正确，再检查集群状态。

 

[root@server05 ~]# grep -n 'master_ip_failover_script' /etc/masterha/app1.cnf

9:master_ip_failover_script= /usr/local/bin/master_ip_failover

Primary Master(192.168.200.111)

 

[root@server01 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

Server05(192.168.200.115)修改故障转移脚本

[root@server05 ~]# head -13 /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip,

$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,

);

my $vip = '192.168.200.100'; # 写入 VIP

my $key = "1"; #非 keepalived 方式切换脚本使用的

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; #若是使用 keepalived

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令

/usr/local/bin/master_ip_failover 文件的内容意思是当主库发生故障时，会触发 MHA 切

换，MHA manager 会停掉主库上的 ens32:1 接口，触发虚拟 ip 漂移到备选从库，从而完成

切换。

Server05(192.168.200.115) 检查 manager 状态

masterha_check_status --conf=/etc/masterha/app1.cnf

app1 is stopped(2:NOT_RUNNING).

 

注意：如果正常会显示"PING_OK"，否则会显示"NOT_RUNNING"，代表 MHA 监控没有开启。

Server05(192.168.200.115) 开启 manager 监控

 

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dea

d_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1

&

[1] 65837

启动参数介绍：

--remove_dead_master_conf 该参数代表当发生主从切换后，老的主库的 ip 将会从配置

文件中移除。

--manger_log 日志存放位置

--ignore_last_failover 在缺省情况下，如果 MHA 检测到连续发生宕机，且两次宕

机间隔不足 8 小时的话，则不会进行 Failover，之所以这样限制是为了避免 ping-pong 效应。

该参数代表忽略上次 MHA 触发切换产生的文件，默认情况下，MHA 发生切换后会在日志目

录，也就是上面我设置的/data 产生 app1.failover.complete 文件，下次再次切换的时候如果

发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方

便，这里设置为--ignore_last_failover。

 

Server05(192.168.200.115)查看 Server05 监控是否正常：

 

[root@monitor ~]# masterha_check_status --conf=/etc/masterha/app1.cnf

app1 (pid:65837) is running(0:PING_OK), master:server01

 

可以看见已经在监控了

 

Server05(192.168.200.115)查看启动日志

[root@server05 ~]# cat /var/log/masterha/app1/manager.log

Sat Dec 29 16:09:50 2018 - [warning] Global configuration file /etc/masterha_default.cnf not

found. Skipping.

Sat Dec 29 16:09:50 2018 - [info] Reading application default configurations from

/etc/masterha/app1.cnf..

Sat Dec 29 16:09:50 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Sat Dec 29 16:09:50 2018 - [info] MHA::MasterMonitor version 0.56.

Sat Dec 29 16:09:51 2018 - [info] Dead Servers:

Sat Dec 29 16:09:51 2018 - [info] Alive Servers:

Sat Dec 29 16:09:51 2018 - [info] server01(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306)

Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306)

Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306)

Sat Dec 29 16:09:51 2018 - [info] Alive Slaves:

Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Primary candidate for the new Master

(candidate_master is set)

Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB 

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB

(oldest major version between slaves) log-bin:enabled

Sat Dec 29 16:09:51 2018 - [info] Replicating from

192.168.200.111(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Current Alive Master: server01(192.168.200.111:3306)

Sat Dec 29 16:09:51 2018 - [info] Checking slave configurations..

Sat Dec 29 16:09:51 2018 - [info] Checking replication filtering settings..

Sat Dec 29 16:09:51 2018 - [info] binlog_do_db= , binlog_ignore_db=

Sat Dec 29 16:09:51 2018 - [info] Replication filtering check ok.

Sat Dec 29 16:09:51 2018 - [info] Starting SSH connection tests..

Sat Dec 29 16:09:58 2018 - [info] All SSH connection tests passed successfully.

Sat Dec 29 16:09:58 2018 - [info] Checking MHA Node version..

Sat Dec 29 16:10:01 2018 - [info] Version check ok.

Sat Dec 29 16:10:01 2018 - [info] Checking SSH publickey authentication settings on the current

master..

Sat Dec 29 16:10:02 2018 - [info] HealthCheck: SSH to server01 is reachable.

Sat Dec 29 16:10:03 2018 - [info] Master MHA Node version is 0.56.

Sat Dec 29 16:10:03 2018 - [info] Checking recovery script configurations on the current

master..

Sat Dec 29 16:10:03 2018 - [info] Executing command: save_binary_logs --command=test

--start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --

manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:10:03 2018 - [info]

Connecting to root@server01(server01)..

Creating /tmp if not exists.. ok.

Checking output directory is accessible or not..

ok.

Binlog found at /var/lib/mysql, up to master-bin.000001

Sat Dec 29 16:10:03 2018 - [info] Master setting check done.

Sat Dec 29 16:10:03 2018 - [info] Checking SSH publickey authentication and checking recovery

script configurations on all alive slave servers..

Sat Dec 29 16:10:03 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:10:03 2018 - [info] Connecting to

root@192.168.200.112(server02:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002

Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:10:04 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:10:04 2018 - [info] Connecting to

root@192.168.200.113(server03:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to slave-relay-bin.000002

Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:10:05 2018 - [info] Executing command : apply_diff_relay_logs

--command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114

--slave_po

rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56

--relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/

--slave_pass=xxxSat Dec 29 16:10:05 2018 - [info] Connecting to

root@192.168.200.114(server04:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to slave-relay-bin.000002

Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002

Testing mysql connection and privileges.. done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Dec 29 16:10:06 2018 - [info] Slaves settings check done.

Sat Dec 29 16:10:06 2018 - [info]

server01 (current master)

+--server02

+--server03

+--server04

Sat Dec 29 16:10:06 2018 - [info] Checking master_ip_failover_script status:

Sat Dec 29 16:10:06 2018 - [info] /usr/local/bin/master_ip_failover --command=status

--ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m

aster_port=3306 

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Checking the Status of the script.. OK

bash: /etc/init.d/keepalived: 没有那个文件或目录

Sat Dec 29 16:10:07 2018 - [info] OK.

Sat Dec 29 16:10:07 2018 - [warning] shutdown_script is not defined.

Sat Dec 29 16:10:07 2018 - [info] Set master ping interval 1 seconds.

Sat Dec 29 16:10:07 2018 - [warning] secondary_check_script is not defined. It is highly

recommended setting it to check master reachability from two or more routes.

Sat Dec 29 16:10:07 2018 - [info] Starting ping health check on

server01(192.168.200.111:3306)..

Thu Aug 31 21:55:23 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't

respond..

注意：其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."说明整个系统已经

开始监控了。

 

关闭 MHA manager 监控，忽略操作

masterha_stop --conf=/etc/masterha/app1.cnf

 

发现已经将 VIP：192.168.200.100 绑定在网卡 ens32。

[root@server01 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

Primary Master(192.168.200.111) 模拟主库故障

[root@server01 ~]# systemctl stop mariadb

[root@server01 ~]# netstat -lnpt | grep :3306

[root@server01 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

slave1(192.168.200.113)状态：

 

MariaDB [(none)]> show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

slave2(192.168.200.114)状态：

MariaDB [(none)]> show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115) 监控已经自动关闭：

 

 

[root@server05 ~]#

[1]+ 完成 nohup masterha_manager --conf=/etc/masterha/app1.cnf

--remove_dead_master_conf --ignore_last_failover < /dev/null >

/var/log/masterha/app1/manager.log 2>&1

 

Server05(192.168.200.115) 查看监控配置文件已经发生了变化（server01 的配置已被删除）：

[root@server05 ~]# cat /etc/masterha/app1.cnf

[server default]

manager_log=/var/log/masterha/app1/manager.log

manager_workdir=/var/log/masterha/app1

master_binlog_dir=/var/lib/mysql

master_ip_failover_script=/usr/local/bin/master_ip_failover

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

user=root

[server2]

candidate_master=1

check_repl_delay=0

hostname=server02

port=3306

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

Server05(192.168.200.115) 故障切换过程中的日志文件内容如下：

[root@server05 ~]# tail -f /var/log/masterha/app1/manager.log

Selected server02 as a new master.

server02: OK: Applying all logs succeeded.

server02: OK: Activated master IP address.

server04: This host has the latest relay log events.

server03: This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

server04: OK: Applying all logs succeeded. Slave started, replicating from server02.

server03: OK: Applying all logs succeeded. Slave started, replicating from server02.

server02: Resetting slave info succeeded.

Master failover to server02(192.168.200.112:3306) completed successfully.

故障主库修复及 VIP 切回测试

Primary Master(192.168.200.111)：

[root@server01 ~]# systemctl start mariadb

[root@server01 ~]# netstat -lnpt | grep :3306

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN

6131/mysqld

Primary Master(192.168.200.111) 指向新的主库

[root@server01 ~]# mysql -u root -p123456

stop slave;

CHANGE MASTER TO

MASTER_HOST='192.168.200.112',

MASTER_USER='repl',

MASTER_PASSWORD='123456';

start slave;

show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.112

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000001

Read_Master_Log_Pos: 1372

Relay_Log_File: mariadb-relay-bin.000002

Relay_Log_Pos: 1208

Relay_Master_Log_File: master-bin.000001

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115) 修改监控配置文件添加 server1 配置：

[root@server05 ~]# vim /etc/masterha/app1.cnf

[server01]

hostname=server01

port=3306

Server05(192.168.200.115) 检查集群状态：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

-----------------------------------忽略部分信息-----------------------------------

Thu Aug 31 22:20:30 2017 - [info] Alive Servers:

Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.111:3306)

Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.112:3306)

Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.113:3306)

Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.114:3306)

-----------------------------------忽略部分信息-----------------------------------

server02 (current master)

+--server01

+--server03

+--server04

-----------------------------------忽略部分信息-----------------------------------

MySQL Replication Health is OK.

Server05(192.168.200.115) 开启监控

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf QQ：981343876 34 / 50 Tel：13651028933

--remove_dead_master_conf --ignore_last_failover<

/dev/null >/var/log/masterha/app1/manager.log 2>&1

&[1] 68199

Secondary Master(192.168.200.112) 关闭现有主库 mysql

[root@server02 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.112/24 brd 192.168.200.255 scope global ens32

inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

[root@server02 ~]# systemctl stop mariadb

[root@server02 ~]# netstat -lnpt | grep :3306

Primary Master(192.168.200.111)

[root@server01 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1

slave1(192.168.200.113)状态：

MariaDB [(none)]> show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.111

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000002

Read_Master_Log_Pos: 923

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

slave2(192.168.200.114)状态：

MariaDB [(none)]> show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.111

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000002

Read_Master_Log_Pos: 923

Relay_Log_File: slave-relay-bin.000002

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115) 配置文件变化（已经移除故障机 server2 配置）：

[root@server05 ~]# cat /etc/masterha/app1.cnf

[server default]

manager_log=/var/log/masterha/app1/manager.log

manager_workdir=/var/log/masterha/app1

master_binlog_dir=/var/lib/mysql

master_ip_failover_script=/usr/local/bin/master_ip_failover

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

user=root

[server1]

hostname=server01

port=3306

[server3]

hostname=server03

port=3306

[server4]

hostname=server04

port=3306

Server05(192.168.200.115) 监控日志：

-----------------------------------忽略部分信息-----------------------------------

Selected server01 as a new master.

server01: OK: Applying all logs succeeded.

server01: OK: Activated master IP address.

server03: This host has the latest relay log events.

server04: This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

server04: OK: Applying all logs succeeded. Slave started, replicating from server01.

server03: OK: Applying all logs succeeded. Slave started, replicating from server01.

server01: Resetting slave info succeeded.

Master failover to server01(192.168.200.111:3306) completed successfully.

修复 Secondary Master(192.168.200.112)主机：

[root@server02 ~]# systemctl start mariadb

[root@server02 ~]# netstat -lnpt | grep :3306

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN

5982/mysqld

Secondary Master(192.168.200.112) 指向新的主库

[root@server02 ~]# mysql -u root -p123456

stop slave;

CHANGE MASTER TO

MASTER_HOST='192.168.200.111',

MASTER_USER='repl',

MASTER_PASSWORD='123456';

start slave;

show slave statusG

MariaDB [(none)]> show slave statusG

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.200.111

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: master-bin.000002

Read_Master_Log_Pos: 923

Relay_Log_File: mariadb-relay-bin.000003

Relay_Log_Pos: 530

Relay_Master_Log_File: master-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Server05(192.168.200.115)修改监控配置文件添加 server2 配置：

[root@server05 ~]# vim /etc/masterha/app1.cnf

[server2]

hostname=server02

candidate_master=1

port=3306

check_repl_delay=0

Server05(192.168.200.115)检查集群状态：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

server01 (current master)

+--server02

+--server03

+--server04

-----------------------------------忽略部分信息-----------------------------------

MySQL Replication Health is OK.

十、配置读请求负载均衡

10.1、安装 LVS 和 Keepalived

192.168.200.116 和 192.168.200.117 主机：

yum -y install ipvsadm kernel-devel openssl-devel keepalived

10.2、master 主机设置

192.168.200.116 lvs master 主机配置：

! Configuration File for keepalived

global_defs {

notification_email {

crushlinux@163.com

}

notification_email_from crushlinux@163.com

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

state MASTER

interface ens32

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.200.200

}

}

virtual_server 192.168.200.200 3306 {

delay_loop 6

lb_algo wrr

lb_kind DR

protocol TCP

real_server 192.168.200.113 3306 {

weight 1

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 3306

}

}

real_server 192.168.200.114 3306 {

weight 1

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 3306

}

}

}

systemctl restart keepalived

检查 VIP 地址

[root@localhost ~]# ip add | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group

default qlen 1000

inet 192.168.200.116/24 brd 192.168.200.255 scope global ens32

inet 192.168.200.200/32 scope global ens32

查看负载策略

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.200.200:3306 wrr

-> 192.168.200.113:3306 Route 1 0 0

-> 192.168.200.114:3306 Route 1 0 0

10.3、backup 主机设置

192.168.200.40 backup 主机配置：

! Configuration File for keepalived

global_defs {

notification_email {

crushlinux@163.com

}

notification_email_from crushlinux@163.com

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

state BACKUP

interface ens32

virtual_router_id 51

priority 99

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.200.200

}

}

virtual_server 192.168.200.200 3306 {

delay_loop 6

lb_algo wrr

lb_kind DR

protocol TCP

real_server 192.168.200.113 3306 {

weight 1

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 3306

}

}

real_server 192.168.200.114 3306 {

weight 1

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 3306

}

}

}

systemctl restart keepalived

10.4、为数据库节点设置 VIP 地址及配置 ARP 相关参数

slave1(192.168.200.113)和 slave2(192.168.200.114)主机配置 realserver 脚本：

[root@server03 ~]# vim realserver.sh

#!/bin/bash

SNS_VIP=192.168.200.200

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP 

/sbin/route add -host $SNS_VIP dev lo:0

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

realserver1 检测 VIP 地址

[root@server03 ~]# bash realserver.sh

[root@server03 ~]# ip a | grep lo

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default

qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0

inet 192.168.200.113/24 brd 192.168.200.255 scope global ens32

inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

远程复制到 realserver2 主机上

[root@server03 ~]# scp realserver.sh 192.168.200.114:/root/

realserver2 检测 VIP 地址

[root@server04 ~]# bash realserver.sh

[root@server04 ~]# ip a |grep lo

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default

qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0

inet 192.168.200.114/24 brd 192.168.200.255 scope global ens32

inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

10.5、数据库读请求测试

Server05(192.168.200.115)担任 client 角色测试，连接 vip

[root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306

Welcome to the MariaDB monitor. Commands end with ; or g.

Your MariaDB connection id is 297

Server version: 5.5.56-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB [(none)]> exit

Bye

断开后在连接测试。。。。

[root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306

Welcome to the MariaDB monitor. Commands end with ; or g.

Your MariaDB connection id is 298

Server version: 5.5.56-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB [(none)]>

查看请求分流

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.200.200:3306 wrr

-> 192.168.200.113:3306 Route 1 1 0

-> 192.168.200.114:3306 Route 1 0 1

[root@localhost ~]# ipvsadm -Lnc

IPVS connection entries

pro expire state source virtual destination

TCP 01:48 FIN_WAIT 192.168.200.115:35462 192.168.200.200:3306

192.168.200.114:3306

TCP 14:50 ESTABLISHED 192.168.200.115:35464 192.168.200.200:3306 192.168.200.113:3306

10.6、数据库连接故障处理

[root@localhost ~]# mysql -u root -p123456 -h192.168.200.113 -P3306

ERROR 1129 (HY000): Host '192.168.200.113' is blocked because of many connection errors;

unblock with 'mysqladmin flush-hosts'

[root@server03 ~]# mysqladmin -uroot -p123456 flush-hosts

附录

手动切换 MHA 脚本

[root@server05 ~]# vim /usr/local/bin/master_ip_online_change

#!/usr/bin/env perl

# Copyright (C) 2011 DeNA Co.,Ltd.

#

# This program is free software; you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation; either version 2 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program; if not, write to the Free Software

# Foundation, Inc.,

# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

## Note: This is a sample script and is not complete. Modify the script based on your

environment.

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

use MHA::DBHelper;

use MHA::NodeUtil;

use Time::HiRes qw( sleep gettimeofday tv_interval );

use Data::Dumper;

my $_tstart;

my $_running_interval = 0.1;

my (

$command, $orig_master_is_new_slave, $orig_master_host,

$orig_master_ip, $orig_master_port, $orig_master_user,

$orig_master_password, $orig_master_ssh_user, $new_master_host,

$new_master_ip, $new_master_port, $new_master_user,

$new_master_password, $new_master_ssh_user

);

my $vip = '192.168.200.100/24'; #VIP

my $key = '1';

my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";

my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down";

my $orig_master_ssh_port = 22;

my $new_master_ssh_port = 22;

GetOptions(

'command=s' => $command,

'orig_master_is_new_slave' => $orig_master_is_new_slave,

'orig_master_host=s' => $orig_master_host,

'orig_master_ip=s' => $orig_master_ip,

'orig_master_port=i' => $orig_master_port,

'orig_master_user=s' => $orig_master_user,

'orig_master_password=s' => $orig_master_password,

'orig_master_ssh_user=s' => $orig_master_ssh_user,

'new_master_host=s' => $new_master_host,

'new_master_ip=s' => $new_master_ip,

'new_master_port=i' => $new_master_port,

'new_master_user=s' => $new_master_user,

'new_master_password=s' => $new_master_password,

'new_master_ssh_user=s' => $new_master_ssh_user,

'orig_master_ssh_port=i' => $orig_master_ssh_port,

'new_master_ssh_port=i' => $new_master_ssh_port,

);

exit &main();

sub current_time_us {

my ( $sec, $microsec ) = gettimeofday();

my $curdate = localtime($sec);

return $curdate . "" . sprintf( "%06d", $microsec );

}

sub sleep_until {

my $elapsed =tv_interval($_tstart);

if ( $_running_interval > $elapsed ) {

sleep( $_running_interval - $elapsed );

}

}

sub get_threads_util {

my $dbh = shift;

my $my_connection_id = shift;

my $running_time_threshold = shift;

my $type = shift;

$running_time_threshold = 0 unless ($running_time_threshold);

$type = 0 unless ($type);

my @threads;

my $sth = $dbh->prepare("SHOW PROCESSLIST");

$sth->execute();

while ( my $ref = $sth->fetchrow_hashref() ) {

my $id = $ref->{Id};

my $user = $ref->{User};

my $host = $ref->{Host};

my $command = $ref->{Command};

my $state = $ref->{State};

my $query_time = $ref->{Time};

my $info = $ref->{Info};

$info =~ s/^s*(.*?)s*$/$1/ if defined($info);

next if ( $my_connection_id == $id );

next if ( defined($query_time) && $query_time < $running_time_threshold );

next if ( defined($command) && $command eq "Binlog Dump" );

next if ( defined($user) && $user eq "system user" );

next

if ( defined($command)

&& $command eq "Sleep"

&& defined($query_time)

&& $query_time >= 1 );

if ( $type >= 1 ) {

next if ( defined($command) && $command eq "Sleep" );

next if ( defined($command) && $command eq "Connect" );

}

if ( $type >= 2 ) {

next if ( defined($info) && $info =~ m/^select/i );

next if ( defined($info) && $info =~ m/^show/i );

}

push @threads, $ref;

}

return @threads;

}

sub main {

if ( $command eq "stop" ) {

## Gracefully killing connections on the current master

# 1. Set read_only= 1 on the new master

# 2. DROP USER so that no app user can establish new connections

# 3. Set read_only= 1 on the current master

# 4. Kill current queries

# * Any database access failure will result in script die.

my $exit_code = 1;

eval {

## Setting read_only=1 on the new master (to avoid accident) 

my $new_master_handler = new MHA::DBHelper();

# args: hostname, port, user, password, raise_error(die_on_error)_or_not

$new_master_handler->connect( $new_master_ip, $new_master_port,

$new_master_user, $new_master_password, 1 );

print current_time_us() . " Set read_only on the new master.. ";

$new_master_handler->enable_read_only();

if ( $new_master_handler->is_read_only() ) {

print "ok. ";

}

else {

die "Failed! ";

}

$new_master_handler->disconnect();

# Connecting to the orig master, die if any database error happens

my $orig_master_handler = new MHA::DBHelper();

$orig_master_handler->connect( $orig_master_ip, $orig_master_port,

$orig_master_user, $orig_master_password, 1 );

## Drop application user so that nobody can connect. Disabling per-session binlog

beforehand

$orig_master_handler->disable_log_bin_local();

print current_time_us() . " Drpping app user on the orig master.. ";

#FIXME_xxx_drop_app_user($orig_master_handler);

## Waiting for N * 100 milliseconds so that current connections can exit

my $time_until_read_only = 15;

$_tstart = [gettimeofday];

my @threads = get_threads_util( $orig_master_handler->{dbh},

$orig_master_handler->{connection_id} );

while ( $time_until_read_only > 0 && $#threads >= 0 ) {

if ( $time_until_read_only % 5 == 0 ) {

printf

"%s Waiting all running %d threads are disconnected.. (max %d milliseconds) ",

current_time_us(), $#threads + 1, $time_until_read_only * 100;

if ( $#threads < 5 ) {

print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . " "

foreach (@threads);

}

}

sleep_until();

$_tstart = [gettimeofday];

$time_until_read_only--;

@threads = get_threads_util( $orig_master_handler->{dbh},

$orig_master_handler->{connection_id} );

}

## Setting read_only=1 on the current master so that nobody(except SUPER) can write

print current_time_us() . " Set read_only=1 on the orig master.. ";

$orig_master_handler->enable_read_only();

if ( $orig_master_handler->is_read_only() ) {

print "ok. ";

}

else {

die "Failed! ";

}

## Waiting for M * 100 milliseconds so that current update queries can complete

my $time_until_kill_threads = 5;

@threads = get_threads_util( $orig_master_handler->{dbh},

$orig_master_handler->{connection_id} );

while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {

if ( $time_until_kill_threads % 5 == 0 ) {

printf

"%s Waiting all running %d queries are disconnected.. (max %d milliseconds) ",

current_time_us(), $#threads + 1, $time_until_kill_threads * 100;

if ( $#threads < 5 ) {

print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . " "

foreach (@threads);

}

}

sleep_until();

$_tstart = [gettimeofday];

$time_until_kill_threads--;

@threads = get_threads_util( $orig_master_handler->{dbh},

$orig_master_handler->{connection_id} );

}

## Terminating all threads

print current_time_us() . " Killing all application threads.. ";

$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );

print current_time_us() . " done. ";

$orig_master_handler->enable_log_bin_local();

$orig_master_handler->disconnect();

## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK

eval {

`ssh -p$orig_master_ssh_port $orig_master_ssh_user@$orig_master_host "

$ssh_stop_vip "`;

};

if ($@) {

warn $@;

} 

$exit_code = 0;

};

if ($@) {

warn "Got Error: $@ ";

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "start" ) {

## Activating master ip on the new master

# 1. Create app user with write privileges

# 2. Moving backup script if needed

# 3. Register new master's ip to the catalog database

# We don't return error even though activating updatable accounts/ip failed so that we don't

interrupt slaves' recovery.

# If exit code is 0 or 10, MHA does not abort

my $exit_code = 10;

eval {

my $new_master_handler = new MHA::DBHelper();

# args: hostname, port, user, password, raise_error_or_not

$new_master_handler->connect( $new_master_ip, $new_master_port,

$new_master_user, $new_master_password, 1 );

## Set read_only=0 on the new master

$new_master_handler->disable_log_bin_local();

print current_time_us() . " Set read_only=0 on the new master. ";

$new_master_handler->disable_read_only();

## Creating an app user on the new master

print current_time_us() . " Creating app user on the new master.. ";

#FIXME_xxx_create_app_user($new_master_handler);

$new_master_handler->enable_log_bin_local();

$new_master_handler->disconnect();

## Update master ip on the catalog database, etc

`ssh -p$new_master_ssh_port $new_master_ssh_user@$new_master_host "

$ssh_start_vip "`;

$exit_code = 0;

};

if ($@) {

warn "Got Error: $@ ";

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "status" ) {

# do nothing

exit 0;

}

else {

&usage();

exit 1;

}

}

sub usage {

print

"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host

--orig_master_ip=ip --orig_master_port=port --new

_master_host=host --new_master_ip=ip --new_master_port=port "; die;

}

[root@server05 ~]# chmod +x /usr/local/bin/master_ip_online_change

 

编辑脚本 send_report

[root@server05 ~]# vim /usr/local/bin/send_report

#!/usr/bin/perl

use strict;

use warnings FATAL => 'all';

use Mail::Sender;

use Getopt::Long;

#new_master_host and new_slave_hosts are set only when recovering master succeeded

my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body );

my $smtp='smtp.163.com';

my $mail_from='xxxx';

my $mail_user='xxxxx';

my $mail_pass='xxxxx';

my $mail_to=['xxxx','xxxx'];

GetOptions(

'orig_master_host=s' => $dead_master_host,

'new_master_host=s' => $new_master_host,

'new_slave_hosts=s' => $new_slave_hosts,

'subject=s' => $subject,

'body=s' => $body,

);

mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body);

sub mailToContacts {

my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_; 

open my $DEBUG, "> /tmp/monitormail.log"

or die "Can't open the debug file:$! ";

my $sender = new Mail::Sender {

ctype => 'text/plain; charset=utf-8',

encoding => 'utf-8',

smtp => $smtp,

from => $mail_from,

auth => 'LOGIN',

TLS_allowed => '0',

authid => $user,

authpwd => $passwd,

to => $mail_to,

subject => $subject,

debug => $DEBUG

};

$sender->MailMsg(

{ msg => $msg,

debug => $DEBUG

}

) or print $Mail::Sender::Error;

return 1;

}

# Do whatever you want here

exit 0;

[root@server05 ~]# chmod +x /usr/local/bin/send_report