HBase Master的高可用实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.HBase集群停止Hmaster观察集群是否可用
1>.停止HMaster服务
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6352 HMaster 5841 DataNode 5731 NameNode 15687 Jps 6168 DFSZKFailoverController 8170 NodeManager 6492 HRegionServer 8060 ResourceManager 5519 QuorumPeerMain hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5474 DataNode 6151 NodeManager 5545 JournalNode 5690 HRegionServer 8140 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5495 DataNode 5628 HRegionServer 6095 NodeManager 8175 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8259 Jps 5700 JournalNode 5624 DataNode 5848 HRegionServer 5501 QuorumPeerMain 6318 NodeManager hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 5651 HRegionServer 8807 Jps 5416 DataNode 5530 JournalNode 6173 NodeManager hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5488 NameNode 5826 HRegionServer 10133 Jps 5708 DFSZKFailoverController 5549 DataNode 6525 NodeManager [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# jps 6352 HMaster 5841 DataNode 5731 NameNode 6168 DFSZKFailoverController 15737 Jps 8170 NodeManager 6492 HRegionServer 8060 ResourceManager 5519 QuorumPeerMain [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hbase-daemon.sh stop master running master, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-master-hadoop101.yinzhengjie.org.cn.out stopping master. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# jps 5841 DataNode 15858 Jps 5731 NameNode 6168 DFSZKFailoverController 8170 NodeManager 6492 HRegionServer 8060 ResourceManager 5519 QuorumPeerMain [root@hadoop101.yinzhengjie.org.cn ~]#
2>.测试HBase集群是否可用
[root@hadoop101.yinzhengjie.org.cn ~]# hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell Use "help" to get list of supported commands. Use "exit" to quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell Version 2.2.4, r67779d1a325a4f78a468af3339e73bf075888bac, 2020年 03月 11日 星期三 12:57:39 CST Took 0.0022 seconds hbase(main):001:0> list TABLE ERROR: KeeperErrorCode = NoNode for /hbase/master #我们在使用HBase 2.2.4版本时，当停止掉HMaster角色后，发现交互式命令行将无法使用...... For usage try 'help "list"' Took 0.0838 seconds hbase(main):002:0>
3>.HMaster高可用的必要性
在HBase中Hmaster负责监控RegionServer的生命周期，均衡RegionServer的负载，如果Hmaster挂掉了，那么整个HBase集群将陷入不健康的状态，并且此时的工作状态并不会维持太久。
经过上述测试，HBase 2.2.4版本中当HMaster角色挂掉后，集群将进入不可用状态，因此在生产环境中我们需要对HBase的Hmaster角色做高可用配置。
二.配置HMaster实战案例
1>.先搭建HBase完全分布式集群
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12239031.html
2>.停止HBase集群
[root@hadoop101.yinzhengjie.org.cn ~]# stop-hbase.sh stopping hbase.......... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8480 Jps 5495 DataNode 6095 NodeManager hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5700 JournalNode 5624 DataNode 8572 Jps 5501 QuorumPeerMain 6318 NodeManager hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 9121 Jps 5416 DataNode 5530 JournalNode 6173 NodeManager hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5474 DataNode 6151 NodeManager 5545 JournalNode 8457 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5841 DataNode 5731 NameNode 16308 Main 6168 DFSZKFailoverController 8170 NodeManager 17994 Jps 8060 ResourceManager 5519 QuorumPeerMain hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5488 NameNode 10449 Jps 5708 DFSZKFailoverController 5549 DataNode 6525 NodeManager [root@hadoop101.yinzhengjie.org.cn ~]#
3>.创建backup-masters文件并分发至集群各个节点
[root@hadoop101.yinzhengjie.org.cn ~]# echo hadoop105.yinzhengjie.org.cn > /yinzhengjie/softwares/hbase/conf/backup-masters [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hbase/conf/backup-masters hadoop105.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/hbase/conf/backup-masters ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hbase/conf/backup-masters] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hbase/conf/backup-masters] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hbase/conf/backup-masters] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hbase/conf/backup-masters] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hbase/conf/backup-masters] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /yinzhengjie/softwares/hbase/conf/backup-masters' hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop105.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]#
4>.单独修改hadoop105.yinzhengjie.org.cn节点的配置文件
[root@hadoop105.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/hbase/conf/hbase-site.xml [root@hadoop105.yinzhengjie.org.cn ~]# [root@hadoop105.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hbase/conf/hbase-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- /** * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ --> <configuration> <!-- ********** HBase核心配置(必须配置部分) ********** --> <property> <name>hbase.rootdir</name> <value>hdfs://yinzhengjie-hdfs-ha/hbase</value> <description>指定Region服务器共享的目录，用来持久存储HBase的数据，URL必须完全正确，其中包含了文件系统的schema。默认值"${hbase.tmp.dir}/hbase"</description> </property> <property> <name>hbase.tmp.dir</name> <value>/yinzhengjie/data/hbase</value> <description>指定Hbase本地文件系统的临时文件存放路径,默认值为"${java.io.tmpdir}/hbase-${user.name}".</description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>指定HBase集群的运行模式，该值为false时，集群为单机模式；该值为true时，集群是分布式模式。如果将该值设置为false，则HBase与Zookeeper的守护进程将运行在一个JVM中。默认值为false</description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>hadoop101.yinzhengjie.org.cn:2181,hadoop102.yinzhengjie.org.cn:2181,hadoop103.yinzhengjie.org.cn:2181</value> <description>配置zookeeper集群地址,不要指定znode路径,HBase会默认将元数据放在根znode</description> </property> <property> <name>hbase.unsafe.stream.capability.enforce</name> <value>false</value> <description>程序WAL依赖于在组件故障期间hsync进行正确操作的能力，但底层文件系统不支持这样做时需要禁用hsync,默认值为"false"</description> </property> <!-- ********** HMaster相关配置 ********** --> <property> <name>hbase.master.info.bindAddress</name> <value>hadoop105.yinzhengjie.org.cn</value> <description>HBase Master 的 Web UI绑定的地址,默认值为"0.0.0.0"</description> </property> <property> <name>hbase.master.port</name> <value>16000</value> <description>HBase Master绑定端口</description> </property> <property> <name>hbase.master.info.port</name> <value>16010</value> <description>HBase Master的Web UI端口，默认值为:"16010",如果不想启动UI实例，则可以将当前参数设置为-1</description> </property> <!-- ********** HRegionServer相关配置 ********** --> <property> <name>hbase.regionserver.port</name> <value>16020</value> <description>HBase RegionServer绑定的端口，默认值为:"16020".</description> </property> <property> <name>hbase.regionserver.info.port</name> <value>16030</value> <description>HBase RegionServer的Web UI端口，默认值为:"16030"设置为-1可以禁用HBase RegionServer的Web UI。</description> </property> <property> <name>hbase.regionserver.info.bindAddress</name> <value>0.0.0.0</value> <description>HBase RegionServer的Web UI地址,默认值为"0.0.0.0"</description> </property> </configuration> [root@hadoop105.yinzhengjie.org.cn ~]#
5>.再次启动HBase集群
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 10289 Jps 9762 JournalNode 9660 DataNode 6173 NodeManager hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10695 Jps 9166 DataNode 6095 NodeManager hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6151 NodeManager 8985 DataNode 9579 Jps 9086 JournalNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9111 DataNode 9212 JournalNode 5501 QuorumPeerMain 6318 NodeManager 9694 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 21792 Jps 19959 DataNode 8170 NodeManager 19834 NameNode 8060 ResourceManager 5519 QuorumPeerMain 20303 DFSZKFailoverController hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11108 DataNode 11033 NameNode 11258 DFSZKFailoverController 6525 NodeManager 11789 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# start-hbase.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] running master, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-master-hadoop101.yinzhengjie.org.cn.out SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] hadoop102.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop102.yinzhengjie.org.cn.out hadoop104.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop104.yinzhengjie.org.cn.out hadoop106.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop106.yinzhengjie.org.cn.out hadoop105.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop105.yinzhengjie.org.cn.out hadoop101.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop101.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn: running regionserver, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-regionserver-hadoop103.yinzhengjie.org.cn.out hadoop105.yinzhengjie.org.cn: SLF4J: Class path contains multiple SLF4J bindings. hadoop105.yinzhengjie.org.cn: SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] hadoop105.yinzhengjie.org.cn: SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] hadoop105.yinzhengjie.org.cn: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. hadoop105.yinzhengjie.org.cn: running master, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-master-hadoop105.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 9762 JournalNode 10345 HRegionServer 9660 DataNode 6173 NodeManager 10557 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9635 HRegionServer 6151 NodeManager 9847 Jps 8985 DataNode 9086 JournalNode hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11146 Jps 10860 HMaster 9166 DataNode 6095 NodeManager 10751 HRegionServer hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9111 DataNode 9751 HRegionServer 9212 JournalNode 9964 Jps 5501 QuorumPeerMain 6318 NodeManager hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 21952 HMaster 22518 Jps 19959 DataNode 8170 NodeManager 19834 NameNode 8060 ResourceManager 5519 QuorumPeerMain 20303 DFSZKFailoverController 22095 HRegionServer hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 12066 Jps 11108 DataNode 11846 HRegionServer 11033 NameNode 11258 DFSZKFailoverController 6525 NodeManager [root@hadoop101.yinzhengjie.org.cn ~]#
6>.访问hadoop101.yinzhengjie.org.cn节点的Hmaster对应的WebUI地址
浏览器访问: http://hadoop101.yinzhengjie.org.cn:16010/master-status
7>.访问hadoop105.yinzhengjie.org.cn节点的Hmaster对应的WebUI地址
三.验证HBase Master的高可用
1>.停止Active HBase Master角色
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9635 HRegionServer 9987 Jps 6151 NodeManager 8985 DataNode 9086 JournalNode hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 21952 HMaster 22868 ZooKeeperMain 19959 DataNode 23033 Jps 8170 NodeManager 19834 NameNode 8060 ResourceManager 5519 QuorumPeerMain 20303 DFSZKFailoverController 22095 HRegionServer hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10100 Jps 9111 DataNode 9751 HRegionServer 9212 JournalNode 5501 QuorumPeerMain 6318 NodeManager hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 9762 JournalNode 10345 HRegionServer 9660 DataNode 10700 Jps 6173 NodeManager hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11290 Jps 10860 HMaster 9166 DataNode 6095 NodeManager 10751 HRegionServer hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11108 DataNode 11846 HRegionServer 11033 NameNode 11258 DFSZKFailoverController 12298 Jps 6525 NodeManager [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# hbase-daemon.sh stop master running master, logging to /yinzhengjie/softwares/hbase/bin/../logs/hbase-root-master-hadoop101.yinzhengjie.org.cn.out stopping master. [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 22868 ZooKeeperMain 19959 DataNode 23271 Jps 8170 NodeManager 19834 NameNode 8060 ResourceManager 5519 QuorumPeerMain 20303 DFSZKFailoverController 22095 HRegionServer hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10048 Jps 9635 HRegionServer 6151 NodeManager 8985 DataNode 9086 JournalNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5329 QuorumPeerMain 9762 JournalNode 10345 HRegionServer 10761 Jps 9660 DataNode 6173 NodeManager hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10163 Jps 9111 DataNode 9751 HRegionServer 9212 JournalNode 5501 QuorumPeerMain 6318 NodeManager hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11414 Jps 10860 HMaster 9166 DataNode 6095 NodeManager 10751 HRegionServer hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 12369 Jps 11108 DataNode 11846 HRegionServer 11033 NameNode 11258 DFSZKFailoverController 6525 NodeManager [root@hadoop101.yinzhengjie.org.cn ~]#
2>.再次访问hadoop105.yinzhengjie.org.cn节点的Hmaster对应的WebUI地址
3>.测试Hbase集群的交互式命令行是否可用
[root@hadoop101.yinzhengjie.org.cn ~]# hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/ha/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/yinzhengjie/softwares/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell Use "help" to get list of supported commands. Use "exit" to quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell Version 2.2.4, r67779d1a325a4f78a468af3339e73bf075888bac, 2020年 03月 11日 星期三 12:57:39 CST Took 0.0024 seconds hbase(main):001:0> list #很明显，当Active HBase Master节点挂掉后，由于有Backup Master的存在，并不会导致集群不可用哟~ TABLE engineer fruit staff1 staff2 staff3 teacher yinzhengjie2020:student yinzhengjie2020:teacher 8 row(s) Took 0.3705 seconds => ["engineer", "fruit", "staff1", "staff2", "staff3", "teacher", "yinzhengjie2020:student", "yinzhengjie2020:teacher"] hbase(main):002:0>