在Spring Boot项目中使用Redis 集群(续)

说明

之前在单机上配置了Redis集群，并在项目中集成了Redis集群。今天在实际的生产环境中还是遇到了很多问题。这里把生产环境的配置过程记录一下。

在生产环境一共三台服务器，每台服务器上存在一个master节点和两个slave节点。这两个slave节点分别从属于另外两台服务器的master节点。这样在最极端的情况下，即使其中两台服务器宕机，也可以保证redis可用。当然，最后是有六台服务器，分别在不通的机柜中，来保证Redis集群最大的可用性。

安装redis

采用源代码安装方式

cd /usr/local/src
wget http://download.redis.io/releases/redis-5.0.3.tar.gz
tar xzf redis-5.0.3.tar.gz
cd redis-5.0.3
make PREFIX=/usr/local/redis install

注册redis服务

复制启动脚本到 /etc/rc.d/init.d 目录

cp /usr/local/src/redis-5.0.3/utils/redis_init_script  /etc/rc.d/init.d/redis6379
cp /usr/local/src/redis-5.0.3/utils/redis_init_script  /etc/rc.d/init.d/redis6380
cp /usr/local/src/redis-5.0.3/utils/redis_init_script  /etc/rc.d/init.d/redis6381

之后编辑/etc/rc.d/init.d/redis6379文件

修改为以下内容

#!/bin/sh
# chkconfig: 2345 80 90
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.

### BEGIN INIT INFO
# Provides:     redis_6379
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Redis data structure server
# Description:          Redis data structure server. See https://redis.io
### END INIT INFO

REDISPORT=6379
EXEC=/usr/local/redis/bin/redis-server
CLIEXEC=/usr/local/redis/bin/redis-cli

PIDFILE=/var/run/redis_${REDISPORT}.pid
CONF="/etc/redis/${REDISPORT}.conf"

PASS="password"

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                $CLIEXEC -a $PASS -p $REDISPORT shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
    restart)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                $CLIEXEC -a $PASS -p $REDISPORT shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
                echo "Starting Redis server..."
                $EXEC $CONF
        fi
        ;;
    *)
        echo "Please use start or stop as first argument"
        ;;
esac

之后编辑/etc/rc.d/init.d/redis6380文件和/etc/rc.d/init.d/redis6380文件，其中端口替换为对应的6380和6381端口。

编辑完启动脚本之后，添加redis的配置文件。将redis配置文件拷贝到/etf/redis/端口.conf

mkdir /etc/redis 
cp /usr/local/src/redis-5.0.3/redis.conf /etc/redis/6379.conf
cp /usr/local/src/redis-5.0.3/redis.conf /etc/redis/6380.conf
cp /usr/local/src/redis-5.0.3/redis.conf /etc/redis/6381.conf

接下来为启动脚本添加执行权限

chmod +x /etc/init.d/redis6379
chmod +x /etc/init.d/redis6380
chmod +x /etc/init.d/redis6381

下面设置开机自启

chkconfig --add redis6379
chkconfig --add redis6380
chkconfig --add redis6381

接下来我们配置集群

执行命令

mkdir /usr/local/data/
mkdir /usr/local/data/redis/
mkdir /usr/local/data/redis/6379/
mkdir /usr/local/data/redis/6380/
mkdir /usr/local/data/redis/6381/

我们编辑 /etc/redis/6379.conf将它作为一个master节点

注释掉 bind 这一行

protected-mode no
requirepass password
masterauth password
port 6379
daemonize yes
cluster-enabled yes
cluster-config-file /etc/redis/nodes-6379.conf
cluster-node-timeout 15000
pidfile /var/run/redis_6379.pid
dir /usr/local/data/redis/6379/

然后我们编辑redis6380.conf和redis6381.conf文件，修改的内容和6379.conf修改的内容一致，将6379.conf中的端口号6379替换为相应的端口号。

之后我们重启服务器

reboot

或者使用命令

service redis6379 start
service redis6380 start
service redis6381 start

我们在另外俩台服务器上做相同的操作。

之后，我们登录其中一台服务器，来做集群的最后配置，分配不同主机的slot

vim addnode.sh

addnode.sh内容如下

/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 111.111.111.111 6379
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 112.112.112.112 6379
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 113.113.113.113 6379

/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 111.111.111.111 6380
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 112.112.112.112 6380
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 113.113.113.113 6380

/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 111.111.111.111 6381
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 112.112.112.112 6381
/usr/local/redis/bin/redis-cli -h 111.111.111.111 -p 6379 -a password cluster meet 113.113.113.113 6381

编辑完内容之后执行下面的命令

chmod +x addnode.sh && ./addnode.sh

添加完节点之后，我们分配slot。

vim addslots.sh

addslots.sh内容如下

host=$1
port=$2
pwd=$3
start=$4
end=$5
 
for slot in `seq ${start} ${end}`  
do  
    echo "add slot:${slot}"  
    redis-cli -h ${host}  -p ${port} -c -a ${pwd} cluster addslots ${slot}
done

编辑完之后保存退出。继续执行以下的命令

chmod +x addnode.sh 
./addslots.sh 111.111.111.111 6379 password 0       5460 
./addslots.sh 112.112.112.112 6379 password 5461    10922
./addslots.sh 113.113.113.113 6379 password 10923   16383

进入redis-cli终端，通过 cluster node的命令查看节点信息

redis-cli -h 111.111.111.111 -p 6379 -a password -c cluster nodes

输出内容文

23c255***511ee7 111.111.111.111:6379@16379 myself,master - 0 1551942837000 1 connected 0-5460
29574b***e7078c 111.111.111.111:6380@16380 master - 0 1551942834000 3 connected
9c9ebe***7cc875 111.111.111.111:6381@16381 master - 0 1551942835000 0 connected

5dff38***f7421c 112.112.112.112:6379@16379 master - 0 1551942836000 6 connected 5461-10922
45e2b6***8baa79 112.112.112.112:6380@16380 master - 0 1551942837003 4 connected
236e69***baea4e 112.112.112.112:6381@16381 master - 0 1551942835000 7 connected

3786ac***be616c 113.113.113.113:6379@16379 master - 0 1551942835003 2 connected 10923-16383
22ed09***92fc7e 113.113.113.113:6380@16380 master - 0 1551942838004 5 connected
8d8fac***0a0d43 113.113.113.113:6381@16381 master - 0 1551942836003 8 connected

之后我们配置它们之间的主从关系

redis-cli -c -a password -h 111.111.111.111 -p 6380 cluster replicate 5dff38***f7421c
redis-cli -c -a password -h 111.111.111.111 -p 6381 cluster replicate 3786ac***be616c

redis-cli -c -a password -h 112.112.112.112 -p 6380 cluster replicate 23c255***511ee7
redis-cli -c -a password -h 112.112.112.112 -p 6381 cluster replicate 3786ac***be616c

redis-cli -c -a password -h 113.113.113.113 -p 6380 cluster replicate 23c255***511ee7
redis-cli -c -a password -h 113.113.113.113 -p 6381 cluster replicate 5dff38***f7421c

到这时redis集群就配置完成了。

上面的输出中，Redis节点的ID较长，为了现实方便，我在贴代码的时候把区间一段替换成了***。