一、原理说明
官网地址：http://activemq.apache.org/replicated-leveldb-store
原理图：
原理说明：
1）使用Zookeeper集群注册所有的ActiveMQ Broker，但只有其中一个Broker可以提供服务，它将被视为Master,其他的Broker处于待机状态被视为Slave。如果Master因故障而不能提供服务，Zookeeper会从Slave中选举出一个Broker充当Master。
2）Slave连接Master并同步他们的存储状态，Slave不接受客户端连接。所有的存储操作都将被复制到连接至Maste的Slaves。
3）如果Master宕机得到了最新更新的Slave会变成Master。故障节点在恢复后会重新加入到集群中并连接Master进入Slave模式。
4）所有需要同步的消息操作都将等待存储状态被复制到其他法定节点的操作完成才能完成。所以，如给你配置了replicas=3，name法定大小是（3/2）+1 = 2。Master将会存储更新然后等待（2-1）=1个Slave存储和更新完成，才汇报success。有一个node要作为观察者存在。当一个新的Master被选中，你需要至少保障一个法定mode在线以能够找到拥有最新状态的ode，这个node才可以成为新的Master。因此，推荐运行至少3个replica nodes以防止一个node失败后服务中断。
二、集群规划
| 主机 | zookeeper集群端口 | zookeeper安装目录 | AMQ集群bind端口 | AMQ消息tcp端口 | AMQ管理控制台端口 | AMQ安装目录 |
| 10.0.0.11 | 2181 | /data/app/zookeeper | bind=”tcp://0.0.0.0:63631” | 61616 | 8161 | /data/app/activemq |
| 10.0.0.12 | 2181 | /data/app/zookeeper | bind=”tcp://0.0.0.0:63632” | 61616 | 8162 | /data/app/activemq |
| 10.0.0.13 | 2181 | /data/app/zookeeper | bind=”tcp://0.0.0.0:63633” | 61616 | 8163 | /data/app/activemq |
三、zookeeper集群
参考文档：https://www.cnblogs.com/hujinzhong/p/11825535.html
1）安装jdk8
[root@activemq01 ~]# rpm -ivh jdk-8u181-linux-x64.rpm [root@activemq01 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
2）解压zookeeper安装包至指定目录
[root@activemq01 ~]# tar xf zookeeper-3.3.6.tar.gz -C /data/app/ [root@activemq01 ~]# mv /data/app/zookeeper-3.3.6/ /data/app/zookeeper
3）配置文件修改
[root@activemq01 conf]# pwd /data/app/zookeeper/conf [root@activemq01 conf]# cp zoo_sample.cfg zoo.cfg [root@activemq01 conf]# vim zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. dataDir=/opt/zookeeper # the port at which the clients will connect clientPort=2181 #创建数据目录 [root@activemq01 conf]# mkdir -p /opt/zookeeper
4）启动zookeeper并查看状态
[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh start JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@activemq01 ~]# jps 17477 QuorumPeerMain 17498 Jps [root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: standalone [root@activemq01 ~]# netstat -lntp|grep 2181 tcp6 0 0 :::2181 :::* LISTEN 17477/java
5）其他两台做相同配置
[root@activemq02 conf]# ../bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: standalone [root@activemq02 conf]# netstat -lntp|grep 2181 tcp6 0 0 :::2181 :::* LISTEN 16530/java [root@active03 conf]# ../bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: standalone [root@active03 conf]# netstat -lntp|grep 2181 tcp6 0 0 :::2181 :::* LISTEN 17204/java
6）创建myid文件
#三台不一样 [root@activemq01 ~]# echo 1 > /opt/zookeeper/myid [root@activemq02 ~]# echo 2 > /opt/zookeeper/myid [root@active03 ~]# echo 3 > /opt/zookeeper/myid
7）修改配置文件
#三台一样 [root@activemq01 ~]# vim /data/app/zookeeper/conf/zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. dataDir=/opt/zookeeper # the port at which the clients will connect clientPort=2181 #cluster server.1=10.0.0.11:2888:3888 server.2=10.0.0.12:2888:3888 server.3=10.0.0.13:2888:3888
8）重启三台zookeeper
[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh restart [root@activemq02 ~]# /data/app/zookeeper/bin/zkServer.sh restart [root@activemq03 ~]# /data/app/zookeeper/bin/zkServer.sh restart #查看三台的状态 [root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: leader [root@activemq02 ~]# /data/app/zookeeper/bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: follower [root@active03 ~]# /data/app/zookeeper/bin/zkServer.sh status JMX enabled by default Using config: /data/app/zookeeper/bin/../conf/zoo.cfg Mode: follower
四、activemq集群
1）下载并解压activemq（三台相同）
[root@activemq01 ~]# tar xf apache-activemq-5.15.9-bin.tar.gz -C /data/app/ [root@activemq01 ~]# mv /data/app/apache-activemq-5.15.9/ /data/app/activemq
2）修改管理控制台端口
[root@activemq01 ~]# cd /data/app/activemq/conf/ [root@activemq01 conf]# vim jetty.xml 109 <property name="host" value="0.0.0.0"/> 110 <property name="port" value="8161"/>
activemq01配置：
activemq02配置：
activemq03配置：
3）3个节点的brokerName一致（activemq.xml）
4）持久化配置
<persistenceAdapter>
<replicatedLevelDB
directory="${activemq.data}/leveldb"
replicas="3"
bind="tcp://0.0.0.0:63631"
zkAddress="10.0.0.11:2181,10.0.0.12:2181,10.0.0.13:2181"
hostname="10.0.0.11"
zkPath="/activemq/leveldb-stores"
/>
</persistenceAdapter>activemq01配置：
activemq02配置：
activemq03配置：
5）启动activemq集群
前提：先启动zookeeper集群
[root@activemq01 activemq]# ./bin/activemq start [root@activemq02 activemq]# ./bin/activemq start [root@activemq03 activemq]# ./bin/activemq start #activemq01 [root@activemq01 app]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1134/sshd tcp6 0 0 :::61613 :::* LISTEN 20689/java tcp6 0 0 :::61614 :::* LISTEN 20689/java tcp6 0 0 :::63631 :::* LISTEN 20689/java tcp6 0 0 :::61616 :::* LISTEN 20689/java tcp6 0 0 :::3888 :::* LISTEN 18386/java tcp6 0 0 :::42611 :::* LISTEN 20689/java tcp6 0 0 :::22 :::* LISTEN 1134/sshd tcp6 0 0 :::1883 :::* LISTEN 20689/java tcp6 0 0 :::8161 :::* LISTEN 20689/java tcp6 0 0 :::33250 :::* LISTEN 18386/java tcp6 0 0 :::2181 :::* LISTEN 18386/java tcp6 0 0 :::5672 :::* LISTEN 20689/java tcp6 0 0 :::2888 :::* LISTEN 18386/java #activemq02 [root@activemq02 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1091/sshd tcp6 0 0 :::3888 :::* LISTEN 17230/java tcp6 0 0 :::35284 :::* LISTEN 19503/java tcp6 0 0 :::22 :::* LISTEN 1091/sshd tcp6 0 0 :::40127 :::* LISTEN 17230/java tcp6 0 0 :::2181 :::* LISTEN 17230/java #activemq03 [root@activemq03 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1089/sshd tcp6 0 0 :::3888 :::* LISTEN 17812/java tcp6 0 0 :::22 :::* LISTEN 1089/sshd tcp6 0 0 :::40284 :::* LISTEN 17812/java tcp6 0 0 :::40066 :::* LISTEN 20141/java tcp6 0 0 :::2181 :::* LISTEN 17812/java
6）登录zookeeper查看activemq集群状态
[root@activemq01 ~]# /data/app/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[activemq, zookeeper]
[zk: 127.0.0.1:2181(CONNECTED) 1] ls /activemq
[leveldb-stores]
[zk: 127.0.0.1:2181(CONNECTED) 2] ls /activemq/leveldb-stores
[00000000002, 00000000000, 00000000001]
[zk: 127.0.0.1:2181(CONNECTED) 3] get /activemq/leveldb-stores/0000000000
00000000002 00000000000 00000000001
[zk: 127.0.0.1:2181(CONNECTED) 3] get /activemq/leveldb-stores/00000000000
{"id":"activemq","container":null,"address":"tcp://10.0.0.11:63631","position":-1,"weight":1,"elected":"0000000000"} #master
cZxid = 0x200000004
ctime = Sat Mar 21 14:43:41 CST 2020
mZxid = 0x20000000c
mtime = Sat Mar 21 14:44:19 CST 2020
pZxid = 0x200000004
cversion = 0
dataVersion = 4
aclVersion = 0
ephemeralOwner = 0x170fbadab630000
dataLength = 116
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 4] get /activemq/leveldb-stores/00000000001
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null} #slave
cZxid = 0x200000007
ctime = Sat Mar 21 14:44:19 CST 2020
mZxid = 0x20000000a
mtime = Sat Mar 21 14:44:19 CST 2020
pZxid = 0x200000007
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x170fbadab630001
dataLength = 89
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 5] get /activemq/leveldb-stores/00000000002 #slave
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null}
cZxid = 0x20000000e
ctime = Sat Mar 21 14:44:31 CST 2020
mZxid = 0x20000000e
mtime = Sat Mar 21 14:44:31 CST 2020
pZxid = 0x20000000e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x170fbadab630002
dataLength = 89
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 6] 6）管理界面访问
此时只能访问master的管理界面，slave的不能提供访问
五、activemq测试故障切换
#关闭master
[root@activemq01 ~]# /data/app/activemq/bin/activemq stop
#查看选举状态
[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[activemq, zookeeper]
[zk: 127.0.0.1:2181(CONNECTED) 1] ls /activemq
[leveldb-stores]
[zk: 127.0.0.1:2181(CONNECTED) 2] ls /activemq/leveldb-stores
[00000000002, 00000000001]
[zk: 127.0.0.1:2181(CONNECTED) 4] get /activemq/leveldb-stores/00000000001
{"id":"activemq","container":null,"address":"tcp://10.0.0.12:63632","position":-1,"weight":1,"elected":"0000000001"} #master
cZxid = 0x200000007
ctime = Sat Mar 21 14:44:19 CST 2020
mZxid = 0x20000001a
mtime = Sat Mar 21 14:57:44 CST 2020
pZxid = 0x200000007
cversion = 0
dataVersion = 6
aclVersion = 0
ephemeralOwner = 0x170fbadab630001
dataLength = 116
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 5] get /activemq/leveldb-stores/00000000002
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null}
cZxid = 0x20000000e
ctime = Sat Mar 21 14:44:31 CST 2020
mZxid = 0x200000017
mtime = Sat Mar 21 14:57:43 CST 2020
pZxid = 0x20000000e
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x170fbadab630002
dataLength = 89
numChildren = 0此时activemq02成为master，登录管理界面测试
重新启动activemq01，将加入集群成为slave