最近看了点redis集群方面的知识点,想自己搭着试试,想了解更多可以阅读redis官方文档。
因为基于docker来操作快速方便。下面都是基于docker-compose来实现redis集群搭建,有关docker知识不太熟悉的,建议先学习下docker以及docker-compose知识。
这里实现的是三主三从集群。
#最终搭建成功后的目录结构如下 [root@new2 docker-redis-cluster]# tree . . ├── docker-compose.yml ├── master1 │ ├── redis.conf │ └── redis.log ├── master2 │ ├── redis.conf │ └── redis.log ├── master3 │ ├── redis.conf │ └── redis.log ├── slave1 │ ├── redis.conf │ └── redis.log ├── slave2 │ ├── redis.conf │ └── redis.log └── slave3 ├── redis.conf └── redis.log
.yml文件使用yaml语言编写,不了解的可以熟知下YAML语法知识。
docker-compose.yml文件内容如下:
version: '3'
services:
master-1:
container_name: master-1
image: redis
command: redis-server /etc/usr/local/redis.conf
network_mode: "host"
tty: true #后台运行不退出
privileged: true #拥有容器内部命令执行的权限
volumes:
- /opt/www/docker-redis-cluster/master1/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/master1/redis.log:/usr/local/redis/logs/redis.log
master-2:
container_name: master-2
command: redis-server /etc/usr/local/redis.conf
image: redis
network_mode: "host"
tty: true #后台运行不退出
privileged: true #拥有容器内部命令执行的权限
volumes:
- /opt/www/docker-redis-cluster/master2/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/master2/redis.log:/usr/local/redis/logs/redis.log
master-3:
container_name: master-3
image: redis
command: redis-server /etc/usr/local/redis.conf
network_mode: "host"
tty: true
privileged: true
volumes:
- /opt/www/docker-redis-cluster/master3/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/master3/redis.log:/usr/local/redis/logs/redis.log
slave-1:
container_name: slave-1
image: redis
command: redis-server /etc/usr/local/redis.conf
network_mode: "host"
tty: true
privileged: true
volumes:
- /opt/www/docker-redis-cluster/slave1/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/slave1/redis.log:/usr/local/redis/logs/redis.log
slave-2:
container_name: slave-2
image: redis
command: redis-server /etc/usr/local/redis.conf
network_mode: "host"
tty: true
privileged: true
volumes:
- /opt/www/docker-redis-cluster/slave2/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/slave2/redis.log:/usr/local/redis/logs/redis.log
slave-3:
container_name: slave-3
image: redis
command: redis-server /etc/usr/local/redis.conf
network_mode: "host"
tty: true
privileged: true
volumes:
- /opt/www/docker-redis-cluster/slave3/redis.conf:/etc/usr/local/redis.conf
- /opt/www/docker-redis-cluster/slave3/redis.log:/usr/local/redis/logs/redis.log
redis.conf文件内容如下(集群里三主端口6379/6380/6381,三从端口6382/6383/6384,这里仅以master1的redis.conf举例,其他五个port注意修改):
port 6379
bind 0.0.0.0
cluster-enabled yes
logfile "/usr/local/redis/logs/redis.log"
文件中的 cluster-enabled 选项用于开实例的集群模式, 而 cluster-conf-file 选项则设定了保存节点配置文件的路径, 默认值为 nodes.conf.节点配置文件无须人为修改, 它由 Redis 集群在启动时创建, 并在有需要时自动进行更新。
要让集群正常运作至少需要三个主节点,不过在刚开始试用集群功能时, 强烈建议使用六个节点: 其中三个为主节点, 而其余三个则是各个主节点的从节点。
注意在各个masterxxxx目录下创建redis.log文件。
准备完毕,运行:
[root@new2 docker-redis-cluster]# docker-compose up -d Creating slave-2 ... done Creating slave-3 ... done Creating master-3 ... done Creating slave-1 ... done Creating master-2 ... done Creating master-1 ... done #使用ps命令查看可见未有运行的docker容器 [root@new2 docker-redis-cluster]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS #使用docker logs master-1查看主服务器master1日志,可见到redis.log无运行权限,从网上查找相关资料,大概说是运行docker的用户和docker容器内部运行redis-server命令的不是同一用户。
#临时给当前docker-redis-cluster目录下所有目录和文件赋予777权限得到解决,当然这不是好的方案,后面再研究下 [root@new2 docker-redis-cluster]# docker logs master-1 *** FATAL CONFIG FILE ERROR (Redis 6.0.7) *** Reading the configuration file, at line 4 >>> 'logfile "/usr/local/redis/logs/redis.log"' Can't open the log file: Permission denied
集群原理:
Redis-cluster把所有的物理节点映射到[0-16383]slot上,cluster 负责维护node<->slot<->value。
Redis集群中内置了 16384 个哈希槽,当需要在 Redis 集群中放置一个 key-value 时,redis 先对key 使用 crc16 算法算出一个结果,然后把结果对 16384 求余数。
这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽,redis 会根据节点数量大致均等的将哈希槽映射到不同的节点。
如果增加新节点D,需要从A、B、C节点上删除一些hash slot给到D。同样地,如果从集群中删除节点A,A上面的hash slots移动到B和C,当节点A上是空的时候就可以将其从集群中完全删除。
因为将hash slots从一个节点移动到另一个节点并不需要停止其它的操作,添加、删除节点以及更改节点所维护的hash slots的百分比都不需要任何停机时间。移动hash slots是并行的,移动hash slots不会影响其它操作。
每个Redis集群中的节点都需要打开两个TCP连接。一个连接用于正常的给Client提供服务,比如6379,还有一个额外的端口(通过在这个端口号上加10000)作为数据端口,比如16379。
第二个端口(本例中就是16379)用于集群总线,这是一个用二进制协议的点对点通信信道。这个集群总线(Cluster bus)用于节点的失败侦测、配置更新、故障转移授权,等等。
客户端从来都不应该尝试和这些集群总线端口通信,它们只应该和正常的Redis命令端口进行通信。注意,确保在你的防火墙中开放着两个端口,否则,Redis集群节点之间将无法通信。
命令端口和集群总线端口的偏移量总是10000。
docker的redis容器已创建好,现在创建集群模式让他们相互通信(因为没有具体指定cluster的各机器主从关系,默认随机分配主从)。
[root@new2 docker-redis-cluster]# docker exec -it master-1 redis-cli --cluster create 127.0.0.1:6
> 127.0.0.1:6380
> 127.0.0.1:6381
> 127.0.0.1:6382
> 127.0.0.1:6383
> 127.0.0.1:6384 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6383 to 127.0.0.1:6379
Adding replica 127.0.0.1:6384 to 127.0.0.1:6380
Adding replica 127.0.0.1:6382 to 127.0.0.1:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
M: fd3e386cc455c45d4d9aed656dc8bcebfb147122 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
M: 33bebb7958fc5bf48038bd9f097bee2ea77274c9 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
S: 375c25f3c7e0094a660ff6a759a12ab016013142 127.0.0.1:6382
replicates 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65
S: 181a6a3a4fe682e59a8cb2b2b8ccae48f8025c81 127.0.0.1:6383
replicates fd3e386cc455c45d4d9aed656dc8bcebfb147122
S: 4aef207dcf5e719fb9c206a7de8eb8f6b0ddfa81 127.0.0.1:6384
replicates 33bebb7958fc5bf48038bd9f097bee2ea77274c9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 181a6a3a4fe682e59a8cb2b2b8ccae48f8025c81 127.0.0.1:6383
slots: (0 slots) slave
replicates fd3e386cc455c45d4d9aed656dc8bcebfb147122
S: 4aef207dcf5e719fb9c206a7de8eb8f6b0ddfa81 127.0.0.1:6384
slots: (0 slots) slave
replicates 33bebb7958fc5bf48038bd9f097bee2ea77274c9
M: 33bebb7958fc5bf48038bd9f097bee2ea77274c9 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 375c25f3c7e0094a660ff6a759a12ab016013142 127.0.0.1:6382
slots: (0 slots) slave
replicates 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65
M: fd3e386cc455c45d4d9aed656dc8bcebfb147122 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
进入master1主机
[root@new2 ~]# redis-cli -p 6379
#写入失败,因为key=a的数据不存在6379主机所在槽点区间,set和get操作均会报错
127.0.0.1:6379> set a 100
(error) MOVED 15495 127.0.0.1:6381
127.0.0.1:6379> get a
(error) MOVED 15495 127.0.0.1:6381
127.0.0.1:6379>
127.0.0.1:6379> set aa 100
OK
127.0.0.1:6379> get aa
"100"
集群服务状态确认
[root@new2 docker-redis-cluster]# redis-cli --cluster check 127.0.0.1:6379
127.0.0.1:6379 (0e4bc440...) -> 1 keys | 5461 slots | 1 slaves.
127.0.0.1:6384 (4aef207d...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6380 (fd3e386c...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 2 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 181a6a3a4fe682e59a8cb2b2b8ccae48f8025c81 127.0.0.1:6383
slots: (0 slots) slave
replicates fd3e386cc455c45d4d9aed656dc8bcebfb147122
M: 4aef207dcf5e719fb9c206a7de8eb8f6b0ddfa81 127.0.0.1:6384
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 33bebb7958fc5bf48038bd9f097bee2ea77274c9 127.0.0.1:6381
slots: (0 slots) slave
replicates 4aef207dcf5e719fb9c206a7de8eb8f6b0ddfa81
S: 375c25f3c7e0094a660ff6a759a12ab016013142 127.0.0.1:6382
slots: (0 slots) slave
replicates 0e4bc44048dc19b1c7f41178e22cc87ccdab7b65
M: fd3e386cc455c45d4d9aed656dc8bcebfb147122 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
使用注意项:
1. 集群环境不能使用select命令,默认使用db0
[root@new2 redis]# redis-cli 127.0.0.1:6379> select 1 (error) ERR SELECT is not allowed in cluster mode