• Nginx+Keepalived 主备高可用 安装与配置


    环境说明:
    操作系统:CentOS6.7 x86_64
    Nginx版本:nginx-1.9.7
    Keepalived版本:keepalived-1.2.24
    主nginx + Keepalived :10.219.24.26
    备nginx + Keepalived :10.219.24.23
    虚拟IP:10.219.24.100
    后端tomcat_1:10.219.24.21:8080
    后端tomcat_2:10.219.24.21:8081

    架构与原理: 前端双 Nginx + Keepalived ,Nginx反向代理到后端tomcat集群实现负载均衡,Keepalived实现集群高可用,
    主nginx故障后虚拟IP自动漂移到备nginx。

    一、 Nginx + Keepalived 安装

    一、1 nginx-1.9.7 编译安装

    下载地址: http://nginx.org/download/nginx-1.9.7.tar.gz

    补充: 安装 nginx之前,需要先安装一些依赖包:gcc、pcre、zlib
    a、nginx gzip模块需要zlib库
    b、nginx rewrite模块需要pcre库
    c、nginx ssl模块需要openssl库

    1、安装必要依赖包
    [root@mysql03 ~]# yum install -y pcre pcre-devel

    centos 6.7 配置 yum 本地源 链接参考: http://blog.csdn.NET/zhang123456456/article/details/56690945
    2、 Nginx安装
    [root@mysql03 ~]# ll nginx-1.9.7.tar.gz
    -rw-r--r--. 1 root root 885562 Jun 14 21:46 nginx-1.9.7.tar.gz
    [root@mysql03 ~]# tar zxvf nginx-1.9.7.tar.gz
    [root@mysql03 ~]# cd nginx-1.9.7
    -- 配置nginx安装选项
    [root@mysql03 nginx-1.9.7]# ./configure --prefix=/usr/local/nginx
    说明: 配置完毕后可以看到一个配置概要,概要中的5项必须都有了相应的库支持
    Configuration summary
    + using system PCRE library
    + OpenSSL library is not used
    ##如果想要安装openssl模块,安装时需指定 ./configure --prefix=/usr/local/nginx --with-openssl=/root/openssl-1.0.2d ##
    + md5: using system crypto library
    + sha1: using system crypto library
    + using system zlib library
    -- 安装nginx
    [root@mysql03 nginx-1.9.7]# make && make install
    3、 检查安装是否正常
    [root@mysql03 nginx-1.9.7]# cd /usr/local/nginx
    [root@mysql03 nginx]# ll
    total 16
    drwxr-xr-x. 2 root root 4096 Jun 14 22:14 conf
    drwxr-xr-x. 2 root root 4096 Jun 14 22:14 html
    drwxr-xr-x. 2 root root 4096 Jun 14 22:14 logs
    drwxr-xr-x. 2 root root 4096 Jun 14 22:14 sbin
    -- 启动
    [root@mysql03 nginx]# ./sbin/nginx #如果不能正常启动,可能是端口占用
    [root@mysql03 nginx]# ps -ef|grep nginx
    root 5212 1 0 22:17 ? 00:00:00 nginx: master process ./sbin/nginx
    nobody 5213 5212 0 22:17 ? 00:00:00 nginx: worker process
    root 5228 2359 0 22:20 pts/0 00:00:00 grep nginx
    -- 访问
    浏览器输入: http://10.219.24.26/ #ip换成自己的ip
    看到以下页面内容,一切正常。
    Welcome to nginx!

    If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

    For online documentation and support please refer to nginx.org.
    Commercial support is available at nginx.com.

    Thank you for using nginx.

    -- 关闭
    [root@mysql03 nginx]# ./sbin/nginx -s stop
    [root@mysql03 nginx]# ps -ef|grep nginx
    root 5241 2359 0 22:25 pts/0 00:00:00 grep nginx

    一、2 安装 keepalived-1.2.24

    一、2.1 安装 LVS

    前提:已经提前配置好本地 Yum 源 配置过程可参考> http://blog.csdn.NET/zhang123456456/article/details/56690945
    1、 检查一下系统内核
    安装之前,首先检查一下系统内核是否支持LVS的IPVS模块;自Linux 2.6开始,系统内核完全内置了LVS的各个模块。
    [root@mysql03 ~]# modprobe -l|grep ipvs
    kernel/net/netfilter/ipvs/ip_vs.ko
    kernel/net/netfilter/ipvs/ip_vs_rr.ko
    kernel/net/netfilter/ipvs/ip_vs_wrr.ko
    kernel/net/netfilter/ipvs/ip_vs_lc.ko
    ....
    kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko
    看到如上输出信息,则表明系统内核默认支持IPVS模块。

    2、安装依赖包
    由于是源码包安装,所以编译时需要打上如下依赖包。
    # yum -y install kernel-headers glibc-headers glibc-devel libgomp gcc libstdc++-devel gcc-c++

    3、 yum安装 ipvsadm-1.26-4
    [root@mysql03 ~]# yum -y install kernel-devel ipvsadm
    ===================================================================================
    Package Arch Version Repository Size
    ===================================================================================
    Installing:
    ipvsadm x86_64 1.26-4.el6 c6-media 42 k

    Transaction Summary
    ===================================================================================
    Install 1 Package(s)

    Total download size: 42 k
    Installed size: 78 k
    Downloading Packages:
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : ipvsadm-1.26-4.el6.x86_64 1/1
    Verifying : ipvsadm-1.26-4.el6.x86_64 1/1

    Installed:
    ipvsadm.x86_64 0:1.26-4.el6

    Complete!
    4、 做个 ln
    [root@mysql03 ~]# ln -sv /usr/src/kernels/2.6.32-573.el6.x86_64/ /usr/src/linux <--请自行修改 kernels 版本号

    5、 查看 ipvsadm 版本信息
    [root@mysql03 ~]# ipvsadm -version
    ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1) #> 能看到帮助信息,则说明IPVS安装成功。


    一、2.2 安装 keepalived-1.2.24

    1、官网下载地址:
    [root@mysql03 ~]# wget http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
    2、解压安装包:
    [root@mysql03 ~]# ll keepalived-1.2.24.tar.gz
    -rw-r--r--. 1 root root 601873 Jun 16 23:49 keepalived-1.2.24.tar.gz
    [root@mysql03 ~]# tar -zxvf keepalived-1.2.24.tar.gz
    3、进入安装目录:
    [root@mysql03 ~]# cd keepalived-1.2.24

    4、安装keepalived:
    [root@mysql03 keepalived-1.2.24]# ./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-573.el6.x86_64/

    说明:--with-kernel-dri :指定使用内核源码中的头文件,即include目录,该参数很重要,只有使用LVS时才用此参数。
    主要输出:
    Keepalived configuration
    ------------------------
    Keepalived version : 1.2.24
    Compiler : gcc
    Preprocessor flags :
    Compiler flags : -Wall -Wunused -Wstrict-prototypes
    Linker flags :
    Extra Lib : -ldl -lssl -lcrypto
    Use IPVS Framework : Yes
    IPVS use libnl : No
    IPVS syncd attributes : No
    IPVS 64 bit stats : No
    fwmark socket support : Yes
    Use VRRP Framework : Yes
    Use VRRP VMAC : Yes
    Use VRRP authentication : Yes
    With ip rules/routes : Yes
    SNMP keepalived support : No
    SNMP checker support : No
    SNMP RFCv2 support : No
    SNMP RFCv3 support : No
    SHA1 support : No
    Use Debug flags : No
    Stacktrace support : No
    Memory alloc check : No
    libnl version : None
    Use IPv4 devconf : No
    Use libiptc : No
    Use libipset : No
    Build genhash : Yes
    Build documentation : No
    [root@mysql03 keepalived-1.2.24]# make && make install

    5、复制脚本和文件:

    5.1 拷贝执行文件
    [root@mysql03 keepalived-1.2.24]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
    5.2 将init.d文件拷贝到etc下,加入开机启动项
    [root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
    5.3 将keepalived文件拷贝到etc下,加入网卡配置
    [root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
    5.4 创建keepalived文件夹
    [root@node6 keepalived-1.2.16]# mkdir /etc/keepalived
    5.5 将keepalived配置文件拷贝到etc下
    [root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
    5.6 复制启动脚本
    [root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/sbin/keepalived /etc/keepalived/

    6、加入开机启动项
    [root@mysql03 keepalived-1.2.24]# chkconfig --add keepalived #添加时必须保证/etc/init.d/keepalived存在
    7、 启动
    [root@mysql03 keepalived-1.2.24]# service keepalived start
    Starting keepalived: [ OK ]
    8、 关闭
    [root@mysql03 keepalived-1.2.24]# service keepalived stop
    Stopping keepalived: [ OK ] > OK,成功!

    说明: 安装完 主nginx + Keepalived 后, 同理安装 备nginx + Keepalived 。

    二、 Nginx + Keepalived 配置

    1. 主 nginx /usr/local/nginx/conf/nginx.conf 配置
    [root@mysql03 ~]# vi /usr/local/nginx/conf/nginx.conf
    [root@mysql03 nginx]# cat /usr/local/nginx/conf/nginx.conf
    user root root;
    worker_processes 1;

    events {
    worker_connections 1024;
    }

    http {
    include mime.types;
    default_type application/octet-stream;

    upstream web_app {
    server 10.219.24.21:8080 weight=1 max_fails=2 fail_timeout=30s;
    server 10.219.24.21:8081 weight=1 max_fails=2 fail_timeout=30s;
    }

    server {
    listen 80;
    server_name aa.com;
    charset utf8;
    location / {
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    proxy_pass http://web_app;
    proxy_redirect default;
    }
    }
    }


    说明: 配置完 主 nginx后 , 同理 , 配置 备 nginx

    2. 主 nginx 的keepalived 配置
    [root@mysql03 nginx]# cat /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived

    global_defs {
    router_id nginx-ha1
    }

    vrrp_script check_nginx {
    # 检查nginx状态的脚本
    script "/data/script/check_nginx.sh"
    interval 2
    }

    vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    nopreempt
    authentication {
    auth_type PASS
    auth_pass 1111
    }
    virtual_ipaddress {
    10.219.24.100
    }
    track_script {
    check_nginx
    }
    }


    3. 备 nginx 的 keepalived 配置
    [root@redis01 ~]# vi /etc/keepalived/keepalived.conf
    [root@redis01 ~]# cat /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived

    global_defs {
    router_id nginx-ha2
    }

    vrrp_script check_nginx {
    script "/data/script/check_nginx.sh"
    interval 2
    }

    vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 80
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 1111
    }
    virtual_ipaddress {
    10.219.24.100
    }
    track_script {
    check_nginx
    }
    }

    补充: 如果开了防火墙,则还需进行防火墙设置,如果没有防火墙设置,则可忽略
    # iptables放行组播地址流量:
    iptables -I INPUT -d 224.0.0.18 -j ACCEPT
    service iptables save

    4. 部署nginx状态检查脚本check_nginx.sh

    4.1 检查nc是否安装 , 没有安装nc包 ,要进行 yum -y install nc 安装 , 不然脚本 check_nginx.sh 中 nc命令用不了。
    [root@mysql03 ~]# rpm -q nc
    nc-1.84-24.el6.x86_64

    4.2 创建脚本目录
    [root@mysql03 ~]# mkdir -p /data/script/
    [root@mysql03 ~]# touch /data/script/check_nginx.sh
    [root@mysql03 ~]# cat /data/script/check_nginx.sh
    #!/bin/bash
    # check nginx server status
    # nginx端口,如果有两个端口,只需要设置为 PORTS="80 81",则下面也应改为"$ret1" != 11
    PORTS="80"

    function check_ports {
    for port in $PORTS;do
    nc -z 127.0.0.1 $port | grep -q succeeded
    [ "${PIPESTATUS[1]}" -eq 0 ] && mark=${mark}1
    done
    # 如果mark值为空说明端口都不通。
    # 如果mark等于1,说明有端口是通的。
    echo $mark
    }

    ret1=$(check_ports)
    # 如果nginx端口不通,会尝试重启一次nginx
    if [ "$ret1" != 1 ];then
    /sbin/service nginx stop
    /sbin/service nginx start
    sleep 1
    ret2=$(check_ports)
    # 如果还是有端口不通,表示nginx服务不正常,则停掉keepalived,使VIP发生切换
    [ "$ret2" != 1 ] && /etc/init.d/keepalived stop
    fi


    5. 给脚本设置可执行权限:
    [root@mysql03 ~]# chmod +x /data/script/check_nginx.sh

    说明: 4. 5.操作都在主、备上都操作

    补充一点:
    如果nginx恢复正常后,keepalived不能自动启动,需要编写一个脚本完成这项工作:判断nginx正常后,拉起keepalived。
    脚本放到cron里每分钟执行。

    6. 开启keepalived的日志 (keepalived默认的日志位置为 /var/log/messages ,最好我们将它单独放置)
    6.1 编辑 /etc/sysconfig/keepalived 将 KEEPALIVED_OPTIONS="-D" 改为 KEEPALIVED_OPTIONS="-D -d -S 0"
    [root@mysql03 ~]# vi /etc/sysconfig/keepalived
    KEEPALIVED_OPTIONS="-D -d -S 0"
    6.2 编辑 /etc/rsyslog.conf
    [root@mysql03 ~]# vi /etc/rsyslog.conf (# 配置文件最后面加上下面一行)
    local0.* /var/log/keepalived.log
    6.3 重启rsyslog:
    [root@mysql03 ~]# service rsyslog restart
    说明:按上面配置后,keepalived会把日志记录到/var/log/keepalived.log。

    说明: 6. 操作在主、备上都操作

    7. 测试
    7.1 启动所有的相关服务
    7.2 查看 master nginx 上的vip ,查看漂浮的vip要用 ip a l ,直接ifconfig是看不到的。
    [root@mysql03 ~]# ip a l
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
    inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
    inet 10.219.24.100/32 scope global eth0 ##### VIP #####
    inet6 fe80::20c:29ff:fe79:f402/64 scope link
    valid_lft forever preferred_lft forever
    7.3 tcpdump 抓包, 发现只有 master ip 10.219.24.26 多播包 ,这也符合 VRRP 工作机制
    [root@mysql03 ~]# tcpdump -n 'host 224.0.0.18'
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    23:07:46.796186 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
    23:07:47.796910 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
    23:07:48.798713 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20

    7.4 master nginx 中 keepalived 关闭 , 发现 vip 漂到了 slave nginx 上 , 前端访问未中断
    -- master nginx
    [root@mysql03 ~]# service keepalived stop
    Stopping keepalived: [ OK ]
    [root@mysql03 ~]# ip a l
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
    inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
    inet6 fe80::20c:29ff:fe79:f402/64 scope link
    valid_lft forever preferred_lft forever

    -- slave nginx
    [root@redis01 ~]# ip a l
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:52:56:60 brd ff:ff:ff:ff:ff:ff
    inet 10.219.24.23/8 brd 10.255.255.255 scope global eth0
    inet 10.219.24.100/32 scope global eth0
    inet6 fe80::20c:29ff:fe52:5660/64 scope link
    valid_lft forever preferred_lft forever

    7.5 再将之前关闭的 keepalived 开启,发现 vip 未漂回来, 正好符合主备为 非抢占式。
    [root@mysql03 ~]# service keepalived start
    Starting keepalived: [ OK ]
    [root@mysql03 ~]# ip a l
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
    inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
    inet6 fe80::20c:29ff:fe79:f402/64 scope link
    valid_lft forever preferred_lft forever

    7.6 将 slave nginx 上的 nginx 关闭, 发现 slave nginx 上的 keepalived 进程也被脚本 kill 了,
    顺利的完成了 vip 漂回 master nginx 上 ,且 前端访问不受影响。
    -- slave nginx
    [root@redis01 ~]# /usr/local/nginx/sbin/nginx -s stop
    [root@redis01 ~]# ps -ef|grep keepalived
    root 4907 2191 0 23:29 pts/0 00:00:00 grep keepalived

    -- master nginx
    [root@mysql03 ~]# ip a l
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
    inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
    inet 10.219.24.100/32 scope global eth0
    inet6 fe80::20c:29ff:fe79:f402/64 scope link
    valid_lft forever preferred_lft forever

  • 相关阅读:
    每日日报
    每日日报
    java笔记
    每日日报
    每日日报
    每日日报
    查看当前mysql时区 并设置为北京时间
    springboot 指定配置文件启动, 区分开发和线上分支
    Js Contains方法
    vue $refs的基本用法
  • 原文地址:https://www.cnblogs.com/andy6/p/7048932.html
Copyright © 2020-2023  润新知