centos实现三个节点高可用

centos实现三个节点高可用

使用的资源为keepalived和nginx

高可用主机IP地址
192.168.136.131
192.168.136.133
192.168.136.134

nginx负载均衡配置
192.168.136.131
192.168.136.133
192.168.136.134
三台主机的80端口对应

192.168.136.131:8080
192.168.136.131:8083
192.168.136.134:8080
192.168.136.134:8084
192.168.136.133:8080
192.168.136.133:8081

nginx的Web服务器
192.168.136.131的8080、8083
192.168.136.134的8080、8084
192.168.136.133的8080、8081

1、每个节点安装nginx和安装keepalived

1.1 安装nginx 1.16.1 ,参考
https://www.cnblogs.com/programer-xinmu78/p/11791486.html
1.2 安装keepalived

yum -y install keepalived
启动keepalived

systemctl start keepalived
systemctl enable keepalived

2、修改keepalived配置

2.1 修改master节点的/etc/keepalived/keepalived.conf配置文件信息

! Configuration File for keepalived
vrrp_script check_nginx_alive {
    script "/usr/bin/check_nginx_alive.sh"
    interval 3
    weight -10
}

global_defs {
   router_id lbs_nginx
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.136.125
    }
    track_script {
        check_nginx_alive
    }

}


virtual_server 192.168.136.125 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP
}

2.2 增加master节点的 执行脚本 /usr/bin/check_nginx_alive.sh,无论主节点还是备节点都需要增加该文件

/usr/bin/check_nginx_alive.sh

#!/bin/sh
 
PATH=/bin:/sbin:/usr/bin:/usr/sbin
 
A=`ps -C nginx --no-header |wc -l`
 
if [ $A -eq 0 ]
   then
     echo 'nginx server is died'
     killall keepalived
fi

2.3 修改backup节点的/etc/keepalived/keepalived.conf配置文件信息
注意，不同的备节点的优先级不一样， priority 20

! Configuration File for keepalived
vrrp_script check_nginx_alive {
    script "/usr/bin/check_nginx_alive.sh"
    interval 3
    weight -10
}

global_defs {
   router_id lbs_nginx
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    virtual_router_id 51
    priority 20
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.136.125
    }
    track_script {
        check_nginx_alive
    }

}


virtual_server 192.168.136.125 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP
}

3、 增加nginx的配置，

3.1 修改nginx的配置文件如下

default.conf 8080端口
default_8084.conf 8084端口
default_lbs.conf 负载均衡配置

server {
    listen       8080;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ .php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ .php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /.ht {
    #    deny  all;
    #}
}

3.2 负载均衡端口配置,负载均很名字和要一致，这里是nginx_lbs

upstream nginx_lbs { 
		server 192.168.136.131:8080 weight=1;
		server 192.168.136.131:8083 weight=1;
		server 192.168.136.134:8080 weight=1;
		server 192.168.136.134:8084 weight=1;
		server 192.168.136.133:8080 weight=2;
		server 192.168.136.133:8081 weight=2;
  }


server {
    listen       80;
    server_name  localhost;

    client_max_body_size 50m;
    client_body_buffer_size 256k;
    location / {
       proxy_pass        http://nginx_lbs;
       proxy_set_header X-Forwarded-For $remote_addr;
       proxy_set_header Host $host;
    }
}

4、启动keepalived

4.1 确认nginx的keepalived是否开放
经过查看可以看到131和134的 nginx.conf中的 keepalive_timeout 65;，相当于打开了keepalive

4.2 启动keepalived

启动Master 192.168.136.131和Backup 192.168.136.134 和 Backup 192.168.136.133的 Keepalived节点
systemctl start keepalived

5、查看keepaliced是否已经放开

ip addr 看到地址中是否有浮动地址

http://192.168.136.125可以正常访问了，可以将每个网站下的index.html文件增加相应标识

6、异常情况查看

6.1 每台主机只能访问本主机所有网站和其他主机个别网站，后来发现原因为，防火墙未打开
修改iptables后正常，
命令为：

            vim /etc/sysconfig/iptables 
        添加8080和8083端口 
            -A INPUT -m state –state NEW -m tcp -p tcp –dport 8080 -j ACCEPT 
            -A INPUT -m state –state NEW -m tcp -p tcp –dport 8083 -j ACCEPT
        启动IPTABLES
            systemctl enable iptables.service
            systemctl start iptables.service
        #重启防火墙使配置文件生效 
            systemctl restart iptables.service

6.2 怎么确认目前的通信信息正常
查看网卡接收的信息
安装tcpdump
yum install -y tcpdump
查看端口vrrp信息
tcpdump -i ens32 vrrp -n
正常情况下，只有一个IP会出现在信息中

6.3 查看keepalived的运行日志
tail -f /var/log/messages

6.4 tcpdump命令看到多个IP的信息，并且主备keepalived服务器上的虚拟IP都增加上了
经过核实原因keepalived需要单独的端口进行通信，目前使用112
通过iptables增加后正常
-A INPUT -p 112 -j ACCEPT

6.5 /etc/keepalived/keepalived.conf 设置中不能增加real_server这个参数，增加real_server后报错，估计目前使用广播和组播进行确认，不需要设置具体的IP地址

估计是哪个版本开始就不用设置了或者换了参数了