http://luyx30.blog.51cto.com/1029851/1350832
参考资料:
http://patrick-tang.blogspot.com/2012/06/redis-keepalived-failover-system.html
http://deidara.blog.51cto.com/400447/302402
http://my.oschina.net/guol/blog/182491
http://shiguanghui.iteye.com/blog/2001499
背景
目前,Redis集群的官方方案还处在开发测试中,未集成到稳定版中。且目前官方开发中的Redis Cluster提供的功能尚不完善(可参考官方网站或http://www.redisdoc.com/en/latest/topic/cluster-spec.html),在生产环境中不推荐使用。通过调研发现市面上要实现采用单一的IP来访问,大多采用keepalived实现redis的双机热备作为过渡方案。
环境部署
环境介绍:
Master: 192.168.1.218 redis,keepalived
Slave: 192.168.1.219 redis,keepalived
Virtural IP Address (VIP): 192.168.1.220
设计思路:
两个redis server主从备份。提供redis 服务高可用;两个keepalived 服务主从备份,提供VIP 服务的高可用。
1)每台redis server分别有主,从两个配置文件(redis_master.conf, redis_slave.conf),通过启动脚本启动服务,启动脚本会检测这个redis集群中的其他服务器的角色,如果有master 服务存在,则以slave角色启动,否则将自己以master服务器启动;
2) keepalived 监控脚本,定时(频率:每秒一次)检测当前服务器是否获取集群VIP,如果获取集群VIP,则将本服务器上的redis服务器设置为master。同时将远端的其他redis服务器设置为slave;保证获取集群VIP的redis 服务器角色为master,其他的设置为slave。
3) keepalived 监控脚本,还会自动检测当前redis服务器是否正常,如果连续两次检测异常,则停掉本本机的keepalived服务,释放集群VIP,让去漂移到其他可以提供redis 服务的服务器上;
4)当 Master 与 Slave 均运作正常时, Master负责服务,Slave负责同步数据;当 Master 挂掉,Slave 正常时, Slave接管服务,同时关闭主从复制功能;当 Master 恢复正常,则从Slave同步数据,同步数据之后关闭主从复制功能,恢复Master身份,于此同时Slave等待Master同步数据完成之后,恢复Slave身份。然后依次循环。
实施步骤:
----创建专用用户
useradd -g develop redisadmin
echo Hisun@1125|passwd --stdin redisadmin
说明:以下部署过程都是在root(或具备sudo权限的账号)账户下进行。
----安装配置redis
1.下载redis源码
cd
wget http://download.redis.io/releases/redis-2.8.3.tar.gz
2.安装redis
tar -zxvf redis-2.8.3.tar.gz
cd redis-2.8.3
#reds的安装可以不用执行configure
make
#测试
make test
####在速度较慢的机器上执行make test可能出现下列错误,无影响
#*** [err]: Test replication partial resync: no backlog in tests/integration/replication-psync.tcl
3.配置redis
#创建redis主目录
mkdir -p /usr/local/redis-2.8.3/{bin,conf,logs}
cp -a -R -p src/redis-server /usr/local/redis-2.8.3/bin/
cp -a -R -p src/redis-cli /usr/local/redis-2.8.3/bin/
cp -a -R -p src/redis-benchmark /usr/local/redis-2.8.3/bin/
cp -a -R -p src/redis-sentinel /usr/local/redis-2.8.3/bin/
cp -a -R -p src/redis-check-dump /usr/local/redis-2.8.3/bin/
cp -a -R -p src/redis-check-aof /usr/local/redis-2.8.3/bin/
#创建redis启动脚本
vi /usr/local/redis-2.8.3/redis-start.sh
####以下为master上的配置,slave上的配置只需要修改对应的LOCALIP和REMOTEIP即可。
#!/bin/bash REDISPATH=/usr/local/redis-2.8.3 REDISCLI=$REDISPATH/bin/redis-cli LOGFILE=$REDISPATH/logs/redis-state.log LOCALIP=192.168.1.218 REMOTEIP=192.168.1.219 REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"` if grep "role:master" <<< $REMOTEREDISROLE ; then #start as slave $REDISPATH/bin/redis-server $REDISPATH/conf/redis_slave.confif [ "$?" == "0" ];then echo "[INFO]`date +%F/%H:%M:%S` :$LOCALIP start as slave successful." >> $LOGFILEelseecho "[ERROR]`date +%F/%H:%M:%S` :$LOCALIP start as slave error." >> $LOGFILE fi else#start as master $REDISPATH/bin/redis-server $REDISPATH/conf/redis_master.confif [ "$?" == "0" ];then echo "[INFO]`date +%F/%H:%M:%S` :$LOCALIP start as master successful." >> $LOGFILEelseecho "[ERROR]`date +%F/%H:%M:%S` :$LOCALIP start as master error." >> $LOGFILE fi fi
#!/bin/bash REDISPATH=/usr/local/redis-2.8.3 LOGFILE=$REDISPATH/logs/redis-state.log kill -9 `ps -ef|grep '/bin/redis-server'|grep -v grep|awk '{print $2}'` if [ "$?" == "0" ];then echo "[INFO]`date +%F/%H:%M:%S` :redis shutdown completed!" >> $LOGFILE elseecho "[ERROR]`date +%F/%H:%M:%S` :redis is not started." >> $LOGFILE fi
#创建redis配置文件
####192.168.1.218主服务器redis_master.conf对应配置项###### #daemonize no daemonize yes #bind 127.0.0.1 bind 192.168.1.218 logfile "/usr/local/redis-2.8.3/logs/redis.log" #其他配置依据实际生产环境修改 ######################################################## ####192.168.1.219从服务器redis_master.conf对应配置项###### #daemonize no daemonize yes #bind 127.0.0.1 bind 192.168.1.219 logfile "/usr/local/redis-2.8.3/logs/redis.log" #其他配置依据实际生产环境修改 ########################################################
####192.168.1.218主服务器redis_slave.conf对应配置项####### #daemonize no daemonize yes #bind 127.0.0.1 bind 192.168.1.218 logfile "/usr/local/redis-2.8.3/logs/redis.log" # slaveof <masterip> <masterport> slaveof 192.168.1.219 6379 #其他配置依据实际生产环境修改 ######################################################## ####192.168.1.219从服务器redis_slave.conf对应配置项####### #daemonize no daemonize yes #bind 127.0.0.1 bind 192.168.1.219 logfile "/usr/local/redis-2.8.3/logs/redis.log" # slaveof <masterip> <masterport> slaveof 192.168.1.218 6379 #其他配置依据实际生产环境修改 ########################################################
#修改redis的属主和权限
chown –R redisadmin:develop /usr/local/redis-2.8.3/
----安装配置keepalived
1.下载keepalived源码Release 1.2.9
注意:最新版为1.2.10测试过程中出错.
wget http://www.keepalived.org/software/keepalived-1.2.9.tar.gz
2.安装keepalived
需要安装以下依赖包: make gcc libpopt-dev libnl-dev libcurl4-openssl-dev popt openssl
cd
tar –zxvf keepalived-1.2.9.tar.gz
cd keepalived-1.2.9
./configure --prefix=/usr/local/keepalived
make && make install
3.配置keepalived
#在Master和Slave上创建如下配置文件(可根据实际情况调整):
mv /usr/local/keepalived/etc/keepalived/keepalived.conf /usr/local/keepalived/etc/keepalived/keepalived.conf-bak
vim /usr/local/keepalived/etc/keepalived/keepalived.conf
vrrp_script chk_redis {
script "/usr/local/keepalived/etc/keepalived/scripts/chk_redis.sh" ###监控脚本
interval 2 ###监控时间
}
vrrp_instance VI_1 {
state MASTER ###设置为MASTER
interface eth3 ###监控网卡,依据实际情况来定
virtual_router_id 51
priority 101 ###权重值
advert_int 1
authentication {
auth_type PASS ###加密
auth_pass redis ###密码
}
track_script {
chk_redis ###执行上面定义的chk_redis
}
virtual_ipaddress {
192.168.1.220 ###VIP
}
}
在Master和Slave上创建监控Redis的脚本
mkdir /usr/local/keepalived/etc/keepalived/scripts
vi /usr/local/keepalived/etc/keepalived/scripts/chk_redis.sh
####以下为master上的配置,slave上的配置只需要修改对应的LOCALIP和REMOTEIP即可。
#!/bin/bash REDISPATH=/usr/local/redis-2.8.3 REDISCLI=$REDISPATH/bin/redis-cli LOGFILE=$REDISPATH/logs/redis-state.log LOCALIP=192.168.1.218 REMOTEIP=192.168.1.219 VIP=192.168.1.220 VIPALIVE=`ip a | grep "$VIP"` if [ "$VIPALIVE" == "" ]; then echo "[info]:"`date`" keepalived server is pengding or stop" >> $LOGFILE else echo "bbb" >> $LOGFILE #check local service is running if [ "`$REDISCLI –h $LOCALIP –p 6379 PING`" == "PONG" ]; then # check local redis server role. REDISROLE=`$REDISCLI info | grep "role"` if grep "role:slave" <<< $REDISROLE ; then #change local redis server as master echo "[info1]:"`date`" Run SLAVEOF NO ONE cmd ..." >> $LOGFILE $REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1 #change remoting redis server as slave REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"` if grep "role:master" <<< $REMOTEREDISROLE ; then echo "[info2]:"`date`" Run remote server SLAVEOF cmd ..." >> $LOGFILE $REDISCLI -h $REMOTEIP SLAVEOF $LOCALIP 6379 >> $LOGFILE 2>&1 fi else REMOTEREDISROLE=`$REDISCLI -h $REMOTEIP info | grep "role"` if grep "role:master" <<< $REMOTEREDISROLE ; then echo "[info3]:"`date`" Run remote server SLAVEOF cmd ..." >> $LOGFILE $REDISCLI -h $REMOTEIP SLAVEOF $LOCALIP 6379 >> $LOGFILE 2>&1 fi fi else echo "[warn]:"`date`" redis server($LOCALIP) is not health..." >> $LOGFILE sleep 1 if [ "`$REDISCLI –h $LOCALIP –p 6379 PING`" != "PONG" ]; then echo "[error]:"`date`" redis server($LOCALIP) will be stop..." >> $LOGFILE service keepalived stop fi fi fi
#首先在2台服务器上设置keepalived的启动文件:
cp -a -R -p /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/keepalived
chmod 750 /etc/rc.d/init.d/keepalived
chown root /etc/rc.d/init.d/keepalived
#然后在2台服务器上创建配置文件的链接:
mkdir /etc/keepalived/
ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
系统测试
注意:一定要先启动redis,再启动keealived,否则redis_check.sh会将keepalived自动关闭。
脚本创建完成以后,我们开始按照如下流程进行测试:
1.启动Master上的Redis
/usr/local/redis-2.8.3/redis-start.sh
#关闭时,直接杀死进程或执行以下脚本
/usr/local/redis-2.8.3/redis-stop.sh
2.启动Slave上的Redis
/usr/local/redis-2.8.3/redis-start.sh
#关闭时,直接杀死进程或执行以下脚本
#/usr/local/redis-2.8.3/redis-stop.sh
3.启动Master上的Keepalived
/etc/rc.d/init.d/keepalived start
#关闭方法
#/etc/rc.d/init.d/keepalived stop
4.启动Slave上的Keepalived
/etc/rc.d/init.d/keepalived start
#关闭方法
#/etc/rc.d/init.d/keepalived stop
继续优化中,未完结。