开发运维日记(二) keepalived实现VIP和nginx可用性检查

上篇中已经搭建了Nginx集群，但还缺少高可用的方案。开源的组件keepalived可以方便发送心跳监控Nginx并实现VIP，所有最终选择keepalived实现高可用。

1、安装环境：centos7 、keepalived-1.3.5

2、主机：192.168.30.7 从机：192.168.30.7 VIP：192.168.30.77

 由于主从服务器配置一样，只演示一台机器的配置。

1、安装keepalived（使用源码安装）

---

 ==> 安装库依赖

yum install -y libnl*
yum install -y libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh

==> 编译源码并安装（将keepalived-1.3.5.tar.gz下载至当前目录）

tar xvf keepalived-1.3.5.tar.gz
cd keepalived-1.3.5
./configure --prefix=/usr/local/keepalived
make && make install

 此时会在对应的路径生产以下的文件

（1）/usr/local/etc/keepalived/keepalived.conf

（2）/usr/local/etc/sysconfig/keepalived

（3）/usr/local/sbin/keepalived

 ==> 初始化配置，设置为系统服务

（1）keepalived启动脚本变量引用文件，copy至默认文件路径是/etc/sysconfig/，也可以不做软链接，直接修改启动脚本中文件路径即可（安装目录下）

cp /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/keepalived

 （2）将keepalived主程序加入到环境变量（安装目录下）

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived

 （3）keepalived启动脚本（源码目录下），放到/etc/init.d/目录下就可以使用service命令便捷调用

cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived  /etc/init.d/keepalived

 （4）将配置文件放到默认路径下，此时启动keepalive服务默认读取该配置

mkdir -p /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

 （5）将keepalived设置为系统服务 开机启动

设置为系统服务开机启动

chkconfig --add keepalived
chkconfig keepalived on

 查看开机启动服务

chkconfig –-list

 启动服务|关闭|重启命令

service keepalived start|stop|restart

主从配置一样，这里不赘述

2、配置VIP和nginx可用性检查

---

 ==> 在/etc/keepalived创建 check_nginx.sh

#!/bin/bash
A = `ps aux | grep nginx | grep -v grep | wc -l`
r = 2
if test $[A] -le $[r] 
then
        /usr/local/nginx/sbin/nginx
        sleep 2
        
        re = `ps axu | grep nginx | grep -v grep |wc -l`
        if test $[re] -le $[r] 
        then
              service keepalived stop
        fi
fi

 添加执行权限

chmod 755 /etc/keepalived/check_nginx.sh

==> 修改主节点keepalived.conf（主）

global_defs {
	router_id lb-ecg
	enable_script_security
}
	
vrrp_script chk_nginx {
		script "/etc/keepalived/check_nginx.sh"
		interval 5
		weight -2
}

vrrp_instance VI_1 {
		state MASTER
		interface ens33
		virtual_router_id 240
		mcast_src_ip 192.168.30.7
		priority 100
		advert_int 1
		authentication {
			auth_type PASS
			auth_pass nginx
		}
		virtual_ipaddress {
			192.168.30.77/24
		}
		track_interface {
			ens33
		}
		track_script {
			chk_nginx
		}
}	

 ==> 修改从节点keepalived.conf（从）

global_defs {
	router_id lb-ecg
       enable_script_security
}
	
vrrp_script chk_nginx {
		script "/etc/keepalived/check_nginx.sh"
		interval 5
		weight -2
}
	
vrrp_instance VI_1 {
		state BACKUP
		interface ens33
		virtual_router_id 240
		mcast_src_ip 192.168.30.8
		priority 90
		advert_int 1
		authentication {
		auth_type PASS
		auth_pass nginx
		}
		virtual_ipaddress {
			192.168.30.77/24
		}
		track_interface {
			ens33
		}
		track_script {
			chk_nginx
		}
}

==> 启动keepalived

service keepalived start

 

注意点：

（1）需要打开 enable_script_security 否则会 【security violation - scripts are being executed but script_security not enabled.】

 （2）check_nginx.sh需要赋予755权限否则会 【keepalived unsafe permissions found for script】

3、VIP漂移检查 和 nginx检测

---

 当keepalived、nginx配置完成并启动机器192.168.30.7/8

==> 192.168.30.7机器：此时VIP(192.168.30.77)绑定在192.168.30.7的ens33的网卡上

[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.30.77/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]# 

==> 192.168.30.8机器：未绑定VIP(192.168.30.77)

[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]# 

 ==> 验证性测试：

（1）kill掉192.168.30.7的keepalived，VIP(192.168.30.77)将漂移到192.168.30.8上。重新启动192.168.30.7的keepalived，VIP(192.168.30.77)将漂回192.168.30.7。

结果如下： 192.168.30.7机器

[root@localhost keepalived]# ps aux | grep keepalived
root      49997  0.0  0.0  48088  1048 ?        Ss   00:39   0:00 /usr/local/keepalived/sbin/keepalived -D
root      49998  0.0  0.0  48088  1792 ?        S    00:39   0:00 /usr/local/keepalived/sbin/keepalived -D
root      49999  0.0  0.0  48088  1392 ?        S    00:39   0:00 /usr/local/keepalived/sbin/keepalived -D
root      92683  0.0  0.0  48088   628 ?        S    01:09   0:00 /usr/local/keepalived/sbin/keepalived -D
root      92684  0.0  0.0 115304  1496 ?        S    01:09   0:00 /bin/bash /etc/keepalived/check_nginx.sh
root      92749  0.0  0.0 112712   972 pts/2    R+   01:09   0:00 grep --color=auto keepalived
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.30.77/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]# 
[root@localhost keepalived]# kill -9 49997
[root@localhost keepalived]# ps aux | grep keepalived
root      93225  0.0  0.0 112712   968 pts/2    R+   01:10   0:00 grep --color=auto keepalived
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]#
[root@localhost keepalived]# service keepalived start
Starting keepalived (via systemctl):  ^C
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.30.77/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]# 

 192.168.30.8机器

。。。此时此时192.168.30.7的keepalived已经被kill掉

[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.30.77/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

。。。。 此时192.168.30.7的keepalived重启完毕

[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
[root@localhost keepalived]# 

 （2）kill掉192.168.30.7的Nginx, 由于keepalived定时脚本 Nginx进程将自动恢复

[root@localhost keepalived]# ps aux | grep nginx
root     113806  0.0  0.0 115304  1496 ?        S    01:25   0:00 /bin/bash /etc/keepalived/check_nginx.sh
root     113871  0.0  0.0 112712   964 pts/2    R+   01:25   0:00 grep --color=auto nginx
root     127705  0.0  0.0  20568   672 ?        Ss   00:04   0:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody   127707  0.0  0.0  20960  1636 ?        S    00:04   0:00 nginx: worker process
[root@localhost keepalived]# /usr/local/nginx/sbin/nginx -s stop
[root@localhost keepalived]# ps aux | grep nginx
root     114421  0.0  0.0 115304  1496 ?        S    01:25   0:00 /bin/bash /etc/keepalived/check_nginx.sh
root     114500  0.0  0.0 115304   556 ?        S    01:25   0:00 /bin/bash /etc/keepalived/check_nginx.sh
root     114502  0.0  0.0 112712   940 ?        S    01:25   0:00 grep nginx
root     114506  0.0  0.0 112712   964 pts/2    R+   01:25   0:00 grep --color=auto nginx

。。。等待5秒之后 nginx进程重新启动

[root@localhost keepalived]# ps aux | grep nginx
root     114546  0.0  0.0  20568   676 ?        Ss   01:25   0:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody   114548  0.0  0.0  20960  1388 ?        S    01:25   0:00 nginx: worker process
root     114775  0.0  0.0 115304  1500 ?        S    01:26   0:00 /bin/bash /etc/keepalived/check_nginx.sh
root     114783  0.0  0.0  20564  1640 ?        S    01:26   0:00 /usr/local/nginx/sbin/nginx
root     114813  0.0  0.0 112712   964 pts/2    R+   01:26   0:00 grep --color=auto nginx
[root@localhost keepalived]# 

 查看keepalived运行情况也可验证：

[root@localhost keepalived]# service keepalived status
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2020-01-08 01:13:22 CST; 15min ago
  Process: 96569 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/keepalived.service
           ├─ 96570 /usr/local/keepalived/sbin/keepalived -D
           ├─ 96571 /usr/local/keepalived/sbin/keepalived -D
           ├─ 96572 /usr/local/keepalived/sbin/keepalived -D
           ├─114546 nginx: master process /usr/local/nginx/sbin/nginx
           ├─114548 nginx: worker process
           ├─118238 /usr/local/keepalived/sbin/keepalived -D
           ├─118239 /bin/bash /etc/keepalived/check_nginx.sh
           └─118302 sleep 2

Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.77
Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77
Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77
Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77
Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77
Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) timed out
Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: /etc/keepalived/check_nginx.sh exited due to signal 15
Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 100 to 98
Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) succeeded
Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 98 to 100
[root@localhost keepalived]#