Nginx入门篇（七）之Nginx+keepalived高可用集群

• 一、keepalived介绍

　　keepalived软件最开始是转为负载均衡软件LVS而设计，用来管理和监控LVS集群系统中各个服务节点的状态，后来又加入了可实现高可用的VRRP功能。所以Keepalived除了能管理LVS以外，还可以作为其他服务（如：Nginx、Haproxy、MySQL）的高可用解决方案的软件。Keepalived是类似工作在lay3、lay4和lay7的交换机制的软件。

　　Keepalived软件是通过VRRP协议实现高可用功能。VRRP（虚拟路由器冗余协议）目的就是为了解决静态路由单点故障的问题，它能够保证当个别节点宕机时，整个网络还可以正常地运行。所以Keepalived一方面有配置管理LVS的功能，还可以对LVS下面的节点进行健康监测，另一方面又可以实现系统网络服务的高可用功能。

• 二、Keepalived的三个功能

1、管理LVS负载均衡软件

2、实现对LVS集群节点健康检查功能

3、作为系统网络服务的高可用功能（重点）

　　Keepalived的作用是检测Web服务器的状态，如果有1台web或MySQL服务器宕机或故障，Keepalived检测到后，会将故障的Web服务器或MySQL服务器从集群当中剔除，而当服务器恢复正常后，Keepalived会自动将剔除的服务器重新加入到集群当中，这些工作无需人工参与，需要人工参与的是服务器故障的修复。

• 三、Keepalived的工作原理　　

　　Keepalived高可用之间是通过VRRP进行通信的。那什么是VRRP协议呢？

　　（1）VRRP，全称Virtual Router Redundancy Protocol，中文为虚拟路由冗余协议，VRRP的出现是为了解决静态路由的单点故障。

　　（2）VRRP是通过一种竞选协议机制来决定将路由任务交给某台VRRP路由器的。

　　（3）VRRP用IP多播的方式（默认多播地址：224.0.0.18）实现高可用对之通信。

　　（4）工作做时，主节点发包，备用节点接包，当备用节点接收不到主节点发送的数据包时，就会启动接管程序接管主节点的资源。备用节点可以有多个，通过优先级竞选，但一般的Keepalived系统运行工作中都是一对。

　　（5）VRRP使用了加密协议加密数据，但是目录官方还是推荐以明文的方式配置认证类型和密码。

　　明确了VRRP协议，再看Keepalived工作原理：

　　Keepalived高可用对之间是通过VRRP进行通信，VRRP通过竞选机制来确定主备，主的优先级高于备，因此工作时，主会优先获得所有资源，备节点处于等待状态，当主宕机后，备用节点则会接管主节点资源，然后顶替主节点对外提供服务。

　　在Keepalived服务对之间，只有作为主的服务器会一直发送VRRP广播包，告诉备用节点主节点还活着，此时备用节点不会抢占主，当主不可用时，即备监听不到主发送的广播包时，就会启动相关的服务接管资源，保证业务的连续性，接管速度最快可以小于1秒。

• 四、Keepalived高可用服务部署

1、环境说明

Hostname	IP	角色说明
lb01	192.168.56.12	keepalived  MASTER
lb02	192.168.56.13	keepalived  BACKUP

2、部署Keepalived

（1）安装keepalived 

[root@lb01 ~]# yum install -y keepalived
[root@lb02 ~]# yum install -y keepalived
[root@lb01 ~]# rpm -qa keepalived
keepalived-1.3.5-6.el7.x86_64
[root@lb02 ~]# rpm -qa keepalived
keepalived-1.3.5-6.el7.x86_64

（2）keepalived.conf配置文件高可用部分解析

[root@lb01 ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {    #定义服务故障报警的E-mail地址，可配多个地址，可选配置
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc    #指定发送邮件的发件人，即发送人地址，可选配置
   smtp_server 192.168.200.1    #指定发送邮件的smtp服务器，本机开启了sendmail或postfix就可以使用上面的默认地址发送邮件，可选配置
   smtp_connect_timeout 30        #链接smtp超时时间，可选配置
   router_id LVS_DEVEL            #Keepalived服务器的路由标识，在同一局域网内该标识具有唯一性
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {        #VRRP实例定义区块，定义了一个VI_1的实例，每个vrrp_instance实例可以认为是Keepalived服务的一个实例或作为一个业务服务，在主节点中有的vrrp_instance实例，备用节点也要存在，这样故障才能接管。
    state MASTER            #定义Keepalived的主备状态，只能有MASTER和BACKUP两种状态，并且状态字符要大写
    interface eth0            #定义Keepalived使用的网卡接口
    virtual_router_id 51    #虚拟路由ID标识，这个标识最好是一个数字，并且唯一。MASTER和BACKUP配置中相同实例的这个id必须一致，否则会脑裂。
    priority 100            #优先级配置，数值越大，实例优先级越高，建议MASTER和BACKUP相差50以上为佳。
    advert_int 1            #同步通知间隔，也就是MASTER和BACKUP之间通信检查时间间隔，单位为秒，默认为1.
    authentication {        #权限认证配置
        auth_type PASS        #认证类型有PASS、AH2中，官方推荐PASS，不超过8个字符，同一实例MASTER和BACKUP使用相同密码才能正常通信。
        auth_pass 1111        #认证密码
    }
    virtual_ipaddress {        #虚拟IP地址，可以配置多个IP地址，每个一行，配置时最好明确指定子网掩码和虚拟IP绑定的网络接口。
        192.168.200.16
        192.168.200.17
        192.168.200.18
    }
}

3、Keepalived高可用服务单实例演示

（1）配置Keepalived主服务器lb01 MASTER

[root@lb01 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@lb01 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
        123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb01
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}
[root@lb01 keepalived]# systemctl start keepalived    //配置完成启动keepalived
[root@lb01 keepalived]# ip addr |grep 192.168.56.20    //查看是否有配置的虚拟IP：192.168.56.20
    inet 192.168.56.20/24 scope global secondary eth0:1

（2）配置Keepalived备服务器lb02 BACKUP

[root@lb02 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@lb02 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
        123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}
[root@lb02 keepalived]# systemctl start keepalived    //配置完成启动keepalived
[root@lb02 keepalived]# ip addr |grep 192.168.56.20    //查看是否有配置的虚拟IP：192.168.56.20，备用服务器查看是不存在虚拟IP的，如果有返回结果，说明脑裂了

（3）高可用主备服务器切换测试

（1）停止主上的keepalived服务，查看lb01和lb02的虚拟ip
[root@lb01 keepalived]# systemctl stop keepalived　　　　 //停止主上的keepalived服务
[root@lb01 keepalived]# ip addr |grep 192.168.56.20  　　//lb01上停止keepalived后，查看lb01上是不存在虚拟ip：192.168.56.20  
[root@lb02 keepalived]# ip addr |grep 192.168.56.20　　  //lb02上可以看到虚拟ip：192.168.56.20，实现了VIP漂移
    inet 192.168.56.20/24 scope global secondary eth0:1

（2）重新启动主上的keepalived服务，查看lb01和lb02的虚拟ip
[root@lb01 keepalived]# systemctl start keepalived　　　　 //重新启动lb01上的keepalived
[root@lb01 keepalived]# ip addr |grep 192.168.56.20　　　　//可以看到虚拟ip又重新回到了lb01上
    inet 192.168.56.20/24 scope global secondary eth0:1
[root@lb02 keepalived]# ip addr |grep 192.168.56.20　　　　//lb02上再查询虚拟ip信息是不存在虚拟ip的

 4、Keepalived双实例双主模式演示

（1）修改lb01和lb02的主配置文件，增加一个实例vrrp_VI2

[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb01
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}

vrrp_instance VI_2 {        //增加一个vrrp实例VI2
    state BACKUP
    interface eth0
    virtual_router_id 56
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.30/24 dev eth0 label eth0:2    //虚拟ip为192.168.56.30
    }
}

[root@lb02 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
        123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}

vrrp_instance VI_2 {        //增加一个vrrp实例VI2
    state MASTER
    interface eth0
    virtual_router_id 56
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.30/24 dev eth0 label eth0:2    //虚拟ip为192.168.56.30
    }
}

（2）在lb01和lb02上分别重启Keepalived服务，观察初始VIP设置情况

[root@lb01 keepalived]# systemctl restart keepalived
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
    inet 192.168.56.20/24 scope global secondary eth0:1

[root@lb02 keepalived]# systemctl restart keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
    inet 192.168.56.30/24 scope global secondary eth0:2

启动lb01的Keepalived服务后，初始状态启动了192.168.56.20这个VIP地址，即由VI_1实例配置的VIP对外提供服务。

启动lb02的Keepalived服务后，初始状态启动了192.168.56.30这个VIP地址，即由VI_2实例配置的VIP对外提供服务。

（3）高可用故障切换测试

[root@lb01 keepalived]# systemctl stop keepalived    //停止lb01的keepalived服务
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"    //在lb01上是无法查看到vip
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"    //在lb02上是可以查看到2个vip地址的
    inet 192.168.56.30/24 scope global secondary eth0:2
    inet 192.168.56.20/24 scope global secondary eth0:1
[root@lb01 keepalived]# systemctl start keepalived        //重新启动lb01上的keepalived服务
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"    //可以看到vip地址192.168.56.20飘移回来了
    inet 192.168.56.20/24 scope global secondary eth0:1

同理测试停止lb02上的keepalived服务查看vip信息

[root@lb02 keepalived]# systemctl stop keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
    inet 192.168.56.20/24 scope global secondary eth0:1
    inet 192.168.56.30/24 scope global secondary eth0:2
[root@lb02 keepalived]# systemctl start keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
    inet 192.168.56.30/24 scope global secondary eth0:2
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
    inet 192.168.56.20/24 scope global secondary eth0:1

• 五、Nginx负载均衡配置Keepalived服务

1、环境说明：

Hostname	IP	角色说明
lb01	192.168.56.12	Nginx+Keepalived（MASTER）
lb02	192.168.56.13	Nginx+Keepalived（BACKUP）
web01	192.168.56.11	web01服务-->Nginx
web02	192.168.0.130	web02服务-->Nginx

2、配置web01和web02

[root@web01 vhosts]# cat www.abc.org.conf 
server {
    listen 80;
    server_name 192.168.56.11;
    root /vhosts/html/www;
    index index.html index.htm index.php;
}
[root@web02 vhosts]# cat www.abc.org.conf 
server {
    listen 8080;
    server_name 192.168.0.130;
    root /vhosts/html/www;
    index index.html index.htm index.php;
}

测试web01和web02的主页，进行区分
[root@localhost vhosts]# curl 192.168.56.11
welcome to 192.168.56.11
[root@localhost vhosts]# curl 192.168.0.130:8080
welcome to use 192.168.0.130

3、在lb01和lb02上配置Nginx负载均衡

[root@lb01 keepalived]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;
    upstream web_server_pool {
        server 192.168.56.11:80 weight=1;
        server 192.168.0.130:8080 weight=1;
    }
    server {
    listen 80;
    server_name 192.168.56.20;     //此处的server_name需要配置VIP的地址
    location / {
        proxy_pass http://web_server_pool;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
    }

}

4、在lb01和lb02上配置Keepalived服务

[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb01
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}

[root@lb02 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
        123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb02
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 dev eth0 label eth0:1
    }
}

注意lb01和lb02中Keepalived配置的不同之处

5、访问测试

直接访问：http://192.168.56.20，可以看到刷新页面，分别得到不同的结果，说明Nginx的负载均衡功能实现了，如图：

再停止lb01上的keepalived，再查看是否能够保持访问

[root@lb01 keepalived]# systemctl stop keepalived
[root@lb01 keepalived]# ip addr |grep "192.168.56.20"
[root@lb02 keepalived]# ip addr |grep "192.168.56.20"　　//可以看到停止lb01上的keepalived后，vip在lb02上
    inet 192.168.56.20/24 scope global secondary eth0:1

再进行访问：http://192.168.56.20，一样可以保持访问结果，这就实现了Keepalived的高可用功能，如图：

6、解决Nginx监控检查的问题

　　按照前面的操作，顺利地实现了Nginx的反向代理和负载均衡，也实现了Keepalived的高可用功能，在默认情况下，Keepalived仅仅在对方机器宕机或者Keepalived服务停止时才会接管也业务，而在实际工作当中，也会有其中一台负载均衡器的Nginx宕机了，而Keepalived服务还在运行，这就会导致用户访问的VIP：192.168.56.20无法找到对应的服务。尝试把lb01的Nginx停止，再查看访问情况

（1）首先先进行访问测试，可以看到都是正常的
[root@localhost vhosts]# curl 192.168.56.20
welcome to 192.168.56.110
[root@localhost vhosts]# curl 192.168.56.20
welcome to use 192.168.0.130

（2）停止lb01上的nginx，查看vip依旧还在lb01上
[root@lb01 keepalived]# systemctl stop nginx
[root@lb01 keepalived]# ip addr |grep "192.168.56.20"
    inet 192.168.56.20/24 scope global secondary eth0:1

（3）再进行测试访问，发现连接被拒绝
[root@localhost vhosts]# curl 192.168.56.20
curl: (7) Failed connect to 192.168.56.20:80; Connection refused

那么，如何解决这种业务服务宕机还可以将IP漂移到备用节点上呢？这就需要Keepalived监测脚本了。首先先写一个脚本，如下：

#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
counter=$(ps -C nginx --no-heading |wc -l)
if [ "${counter}" = "0" ]; then
    systemctl start nginx.service
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        echo "$d nginx was down.keepalived will stop." >> /var/log/check_ng.log
    systemctl stop keepalived
    fi
fi

此处在监测到nginx进程为0时，会重新启动nginx，再进行统计nginx的进程数量，如果依旧为0，则将keepalived服务停止，启用高可用故障切换。实验阶段，为了看到效果，使用一下脚本，只要监测到了nginx进程数为0，即刻停止keepalived服务，脚本如下：

此脚本在lb01和lb02上都需要存在的，脚本路径：/etc/keepalived/check_nginx.sh
[root@lb01 keepalived]# cat check_nginx.sh 
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
counter=$(ps -C nginx --no-heading|wc -l)
if [ $counter -eq 0 ]; then
        echo "$d nginx was down.keepalived will stop." >> /var/log/check_ng.log
        systemctl stop keepalived
fi

再对lb01和lb02的keepalived.conf配置文件进行修改，增加脚本模块：

[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    123456@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb01
}

vrrp_script chk_nginx {　　#定义vrrp脚本，检测nginx进程，此处一定要注意和"{"的空格，如果没有空格，会导致脚本不会执行，切记切记！！！
    script "/etc/keepalived/check_nginx.sh"　　#执行脚本，当Nginx服务有问题，就停掉Keepalived
    interval 2　　　　　　　　#监测的间隔时间为2s
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.56.20/24 
    }
    track_script {
    chk_nginx　　　　#在vrrp实例VI_1启用chk_nginx这个脚本
    }
}

下面测试过程和结果：

（1）在lb01上查看keepalived的vip和进程以及nignx的端口
[root@lb01 keepalived]# !ip
ip addr |grep "192.168.56.20"
    inet 192.168.56.20/24 scope global secondary eth0
[root@lb01 keepalived]# netstat -tulnp |grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      7624/nginx: master  
[root@lb01 keepalived]# ps -ef |grep  keepalived
root      7633     1  0 23:49 ?        00:00:00 /usr/sbin/keepalived -D
root      7634  7633  0 23:49 ?        00:00:00 /usr/sbin/keepalived -D
root      7635  7633  0 23:49 ?        00:00:00 /usr/sbin/keepalived -D

（2）模拟Nginx故障，停止Nginx服务，再查看（1）中的相关信息
[root@lb01 keepalived]# systemctl stop nginx
[root@lb01 keepalived]# !nets
netstat -tulnp |grep nginx
[root@lb01 keepalived]# !ip
ip addr |grep "192.168.56.20"
[root@lb01 keepalived]# ps -ef |grep keepalived
root      7881  5009  0 23:51 pts/1    00:00:00 grep --color=auto keepalived

（3）在lb02上查看VIP信息是否存在，并验证web服务访问是否正常
[root@lb02 keepalived]# !ip
ip addr |grep "192.168.56.20"
    inet 192.168.56.20/24 scope global secondary eth0
[root@localhost vhosts]# curl 192.168.56.20
welcome to use 192.168.0.130
[root@localhost vhosts]# curl 192.168.56.20
welcome to 192.168.56.110

通过上述的脚本监测，可以实现了真正的Nginx+Keepalived高可用故障切换功能。

7、写一个监测Keepalived脑裂的脚本

　　为了防止高可用功能出现脑裂现象，还可以在备用服务器上写一个监测脚本，如果可以ping通主节点并且备用节点有VIP就报警。

 （1）在lb02上写一个监测脚本并执行

[root@lb02 keepalived]# cat check_split_brain.sh 
#!/bin/bash
lb01_vip="192.168.56.20"
lb01_ip="192.168.56.12"
while true
do
    ping -c 2 -W 3 $lb01_ip &>/dev/null
    if [ $? -eq 0 -a `ip add|grep "$lb01_vip"|wc -l` -eq 1 ]
    then
    echo "ha is split brain.warning."
    else
    echo "ha is ok."
    fi
    sleep 3 
done
[root@lb02 keepalived]# sh check_split_brain.sh 
ha is ok.
ha is ok.
ha is ok.

正常情况下，主节点还活着，VIP 192.168.56.20就在主节点上，不会报警，提示：ha is ok

（2）模拟脑裂：停止主节点lb01上的Keepalived，查看lb02上的脚本执行情况

[root@lb01 keepalived]# systemctl stop keepalived
[root@lb02 keepalived]# sh check_split_brain.sh 
ha is ok.
ha is split brain.warning.
ha is split brain.warning.

从上可以看到脚本会报警有脑裂的错误，即可将此叫脚本放在zabbix监控服务当中，实现脑裂报警。