一、为什么要使用负载均衡技术？
1、系统高可用性
2、 系统可扩展性
3、 负载均衡能力
LVS+keepalived能很好的实现以上的要求，LVS提供负载均衡，keepalived提供健康检查，故障转移，提高系统的可用性！采用这样的架构以后很容易对现有系统进行扩展，只要在后端添加或者减少realserver，只要更改lvs的配置文件，并能实现无缝配置变更！
二、LVS+Keepalived介绍
1、 LVS
LVS是一个开源的软件，可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写，意思是Linux虚拟服务器。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；八种调度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。
2、 keepalived
Keepalived 是运行在lvs 之上，它的主要功能是实现真实机的故障隔离及负载均衡器间的失败切换，提高系统的可用性
三、环境：
四台服务器，系统全为CentOS6.8：
192.168.2.203 master lvs+keepalived
192.168.2.202 backup lvs+keepalived
192.168.2.204 web1（nginx）
192.168.2.205 web2 （nginx）
vip：192.168.2.13
其中nginx已预装好，这里不再写搭建过程
四、搭建并配置
1、分别在backup lvs和master lvs上安装lvs
1 root@bogon src]# yum -y install ipvsadm 2 已加载插件：fastestmirror 3 设置安装进程 4 Determining fastest mirrors 5 epel/metalink | 5.4 kB 00:00 6 * base: mirror.lzu.edu.cn 7 ... ... 8 已安装: 9 ipvsadm.x86_64 0:1.26-4.el6 10 11 作为依赖被安装: 12 libnl.x86_64 0:1.1.4-2.el6 13 14 完毕！
2、把ipvsadm模块加载进系统
1 [root@bogon src]# ipvsadm 2 IP Virtual Server version 1.2.1 (size=4096) 3 Prot LocalAddress:Port Scheduler Flags 4 -> RemoteAddress:Port Forward Weight ActiveConn InActConn 5 [root@bogon src]# lsmod | grep ip_vs 6 ip_vs 126897 0 7 libcrc32c 1246 1 ip_vs 8 ipv6 336282 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
3、分别在backup lvs和master lvs上安装keepalived（keepalived官网：http://www.keepalived.org/）
[root@bogon src]# tar zxf keepalived-1.2.24.tar.gz [root@bogon src]# cd keepalived-1.2.24 [root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/ 报错： configure: error: !!! OpenSSL is not properly installed on your system. !!! !!! Can not include OpenSSL headers files. !!!
系统缺少openssl-devel包所致
安装openssl-devel
root@bogon keepalived-1.2.24]# yum -y install openssl-devel
再次编辑安装
[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/ Keepalived configuration ------------------------ Keepalived version : 1.2.24 Compiler : gcc Preprocessor flags : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//include Compiler flags : -Wall -Wunused -Wstrict-prototypes Linker flags : Extra Lib : -ldl -lssl -lcrypto Use IPVS Framework : Yes IPVS use libnl : No IPVS syncd attributes : No IPVS 64 bit stats : No fwmark socket support : Yes Use VRRP Framework : Yes Use VRRP VMAC : Yes Use VRRP authentication : Yes With ip rules/routes : Yes SNMP keepalived support : No SNMP checker support : No SNMP RFCv2 support : No SNMP RFCv3 support : No SHA1 support : No Use Debug flags : No Stacktrace support : No Memory alloc check : No libnl version : None Use IPv4 devconf : No Use libiptc : No Use libipset : No Build genhash : Yes Build documentation : No [root@bogon keepalived-1.2.24]# make && make install
[root@bogon keepalived-1.2.24]# ln -s /usr/local/sbin/keepalived /sbin/ [root@bogon keepalived-1.2.24]# chkconfig --add keepalived [root@bogon keepalived-1.2.24]# chkconfig --level 35 keepalived on
4、配置keepalived
lvs-master的配置文件如下
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { #全局配置部分 # notification_email { #email 通知，基本不用此处所以注释掉 # acassen@firewall.loc # failover@firewall.loc # sysadmin@firewall.loc # } # notification_email_from Alexandre.Cassen@firewall.loc # smtp_server 192.168.200.1 # smtp_connect_timeout 30 router_id LVS_DEVEL # 设置lvs的id，在一个网络内应该是唯一的 vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_instance VI_1 { #vrrp实例定义部分 state MASTER #设置lvs的状态，报错MASTER和BACKUP两种，必须大写 interface eth1 #设置对外服务的接口 virtual_router_id 60 #设置虚拟路由标示，这个标示是一个数字，同一个vrrp实例使用唯一标示 priority 100 #定义优先级，数字越大优先级越高，在一个vrrp——instance下，master的优先级必须大于backup advert_int 1 #设定master与backup负载均衡器之间同步检查的时间间隔，单位是秒 authentication { #设置验证类型和密码 auth_type PASS #主要有PASS和AH两种 auth_pass 1111 #验证密码，同一个vrrp_instance下MASTER和BACKUP密码必须相同 } virtual_ipaddress { #设置虚拟ip地址，可以设置多个，每行一个 192.168.2.13 } } virtual_server 192.168.2.13 80 { #设置虚拟服务器，需要指定虚拟ip和服务端口 delay_loop 3 #健康检查时间间隔 lb_algo rr #负载均衡调度算法 lb_kind DR #负载均衡转发规则 persistence_timeout 50 #设置会话保持时间，对动态网页非常有用 protocol TCP #指定转发协议类型，有TCP和UDP两种 real_server 192.168.2.204 80 { #配置服务器节点1，需要指定real server的真实IP地址和端口 weight 1 #设置权重，数字越大权重越高 TCP_CHECK { #realserver的状态监测设置部分单位秒 connect_timeout 3 #超时时间 nb_get_retry 3 #重试次数 delay_before_retry 3 #重试间隔 connect_port 80 #监测端口 } } real_server 192.168.2.205 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } }
LVS-backup的配置文件如下
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { # notification_email { # acassen@firewall.loc # failover@firewall.loc # sysadmin@firewall.loc # } # notification_email_from Alexandre.Cassen@firewall.loc # smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_instance VI_1 { state BACKUP interface eth1 virtual_router_id 60 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.2.13 } } virtual_server 192.168.2.13 80 { delay_loop 3 lb_algo rr lb_kind DR persistence_timeout 3 protocol TCP real_server 192.168.2.204 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 192.168.2.205 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } }
5、realserver的配置
两台web服务器都要执行下面脚本
[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh #!/bin/bash # description: Config realserver lo and apply noarp SNS_VIP=192.168.2.13 /etc/rc.d/init.d/functions case "$1" in start) ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP /sbin/route add -host $SNS_VIP dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce sysctl -p >/dev/null 2>&1 echo "RealServer Start OK" ;; stop) ifconfig lo:0 down route del $SNS_VIP >/dev/null 2>&1 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce echo "RealServer Stoped" ;; *) echo "Usage: $0 {start|stop}" exit 1 esac exit 0
[root@bogon www]# /etc/rc.d/init.d/realserver.sh start /etc/rc.d/init.d/realserver.sh: line 6: /etc/rc.d/init.d/functions: 权限不够 RealServer Start OK [root@bogon www]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:41:71:DF inet addr:192.168.12.129 Bcast:192.168.12.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:728 errors:0 dropped:0 overruns:0 frame:0 TX packets:98 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:137311 (134.0 KiB) TX bytes:7369 (7.1 KiB) eth1 Link encap:Ethernet HWaddr 00:0C:29:41:71:E9 inet addr:192.168.2.204 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:119838 errors:0 dropped:0 overruns:0 frame:0 TX packets:31612 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23411786 (22.3 MiB) TX bytes:2119106 (2.0 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:182 (182.0 b) TX bytes:182 (182.0 b) lo:0 Link encap:Local Loopback inet addr:192.168.2.13 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:65536 Metric:1
6、启动keepalived并进行测试
[root@bogon keepalived-1.2.24]# service keepalived start 正在启动 keepalived： [确定]
lvs-master
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:46 bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP 192.168.2.13 added Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@bogon keepalived-1.2.24]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.2.13:80 rr persistent 3 -> 192.168.2.204:80 Route 1 0 0 -> 192.168.2.205:80 Route 1 0 0
访问curl http://192.168.2.13/test.txt
[root@www etc]# curl http://192.168.2.13/test.txt it is web2 [root@www etc]# curl http://192.168.2.13/test.txt it is web2
关掉web2再次测试
[root@www etc]# curl http://192.168.2.13/test.txt it is web1 [root@www etc]# curl http://192.168.2.13/test.txt it is web1
查看lvs-master
[root@bogon keepalived-1.2.24]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.2.13:80 rr persistent 3 -> 192.168.2.204:80 Route 1 0 2 [root@bogon keepalived-1.2.24]# tail -f /var/log/messages Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80 Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected. Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.
已经自动把web2剔除
打开web2再次查看
[root@bogon keepalived-1.2.24]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.2.13:80 rr persistent 3 -> 192.168.2.204:80 Route 1 0 0 -> 192.168.2.205:80 Route 1 0 0 [root@bogon keepalived-1.2.24]# tail -f /var/log/messages Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry. Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80 Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected. Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25. Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success. Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80 to VS [192.168.2.13]:80 Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
恢复后已自动添加进来
关掉lvs master的keepalived
[root@bogon keepalived-1.2.24]# service keepalived stop 停止 keepalived： [确定]
访问web并查看lvs backup
[root@www etc]# curl http://192.168.2.13/test.txt it is web2 [root@www etc]# curl http://192.168.2.13/test.txt it is web2
[root@lys2 src]# tail -f /var/log/messages Oct 23 19:03:26 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs. Oct 23 19:03:27 lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP 192.168.2.13 added Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13 Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13 Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@lys2 src]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff inet 192.168.144.101/24 brd 192.168.144.255 scope global eth0 inet6 fe80::20c:29ff:fe89:fe3/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff inet 192.168.2.202/24 brd 192.168.2.255 scope global eth1 inet 192.168.2.13/32 scope global eth1 inet6 fe80::20c:29ff:fe89:fed/64 scope link valid_lft forever preferred_lft forever
可以看到lvs backup已自动切换成master状态并自动绑定了vip
查看lvs master vip
[root@bogon keepalived-1.2.24]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff inet 192.168.12.128/24 brd 192.168.12.255 scope global eth0 inet6 fe80::20c:29ff:fe55:4d7a/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1 inet6 fe80::20c:29ff:fe55:4d84/64 scope link valid_lft forever preferred_lft forever
已自动解除vip
到处全部结束