Apache Hadoop 2.9.2 的HDFS High Available模式部署

　　　　　　　　　　　　　　Apache Hadoop 2.9.2 的HDFS High Available 模式部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　我们知道，当NameNode进程挂掉后，可以通过SecondName进行补救，我们可以快速进行恢复操作。但是其效率极低，可能等你启动集群成功后，半小时就过去了，会严重影响业务！这个时候我们就需要对NameNode做高可用。我们可以通过双NameNode消除单点故障。

一.HA概述

1>.所谓High Available，简称HA，即高可用（7*24小时不中断服务）。

2>.实现高可用最关键对策略是消除单点故障。HA严格来说应该分成各个组建对HA机制：HDFS对HA和YARN的HA。

3>.Hadoop2.0版本之前，在HDFS集群中NameNode存在单点故障（SPOF）

4>.NameNode主要在以下两个方面影响HDFS集群：

　　第一：NameNode机器发生意外，如宕机，集群将无法使用，知道管理员重启。

　　第二：NameNode机器需要升级，包括软件，硬件升级，此时集群也将无法使用。

　　HDFS HA功能通过配置Active/Standby两个NameNodes实现集群中对NameNode的热备来解决上述问题。如出现故障，如机器崩溃或机器需要升级维护，这时通过此种方式将NameNode很快的切换到另外一台机器。

二.HDFS HA工作机制

1>.HDFS HA工作要点

一.元数据管理方式需要改变：
　　1>.内存中各自保存一份元数据；
　　2>.Edits日志只有Active状态的namenode节点可以做写操作；
　　3>.两个namenode都可以读取edits；
　　4>.共享的edits放在一个共享存储中管理（qjournal和NFS两个主流实现）；


二.需要一个状态管理功能模块
　　实现了一个zkfailover，常驻在每一个namenode所在的节点，每一个zkfailover负责监控自己所在namenode节点，利用zk进行状态标识，当需要进行状态切换时，由zkfailover来负责切换，切换时需要防止brain split现象的发生。


三.必须保证两个NameNode之间能够ssh无密码登录。


四.隔离（Fence）
　　即同一时刻仅仅有一个NameNode对外提供服务

2>.HDFS HA 自动故障转移工作机制

　　我们知道使用命令hdfs haadmin -failover手动进行故障转移，在该模式下，即使 Active NameNode已经失效，系统也不会自动从Active NameNode转移到 Standby NameNode，下面学习如何配置部署HA自动进行故障转移。自动故障转移为HDFS部署增加了两个新组件：ZooKeeper和ZKFailoverController（ZKFC）进程。

　　ZooKeeper是维护少量协调数据，通知客户端这些数据的改变和监视客户端故障的高可用服务。HA的自动故障转移依赖于ZooKeeper的以下功能：
　　　　故障检测：
　　　　　　集群中的每个NameNode在ZooKeeper中维护了一个持久会话，如果机器崩溃，ZooKeeper中的会话将终止，ZooKeeper通知另一个NameNode需要触发故障转移。
　　　　Active NameNode选择：
　　　　　　ZooKeeper提供了一个简单的机制用于唯一的选择一个节点为active状态。如果目前 Active NameNode崩溃，另一个节点（Standby NameNode）可能从ZooKeeper获得特殊的排外锁以表明它应该成为 Active NameNode。


　　ZKFC是自动故障转移中的另一个新组件，是ZooKeeper的客户端，也监视和管理NameNode的状态。每个运行NameNode的主机也运行了一个ZKFC进程，ZKFC负责：
　　　　健康监测：
　　　　　　ZKFC使用一个健康检查命令定期地ping与之在相同主机的NameNode，只要该NameNode及时地回复健康状态，ZKFC认为该节点是健康的。如果该节点崩溃，冻结或进入不健康状态，健康监测器标识该节点为非健康的。
　　　　ZooKeeper会话管理：
　　　　　　当本地NameNode是健康的，ZKFC保持一个在ZooKeeper中打开的会话。如果本地NameNode处于active状态，ZKFC也保持一个特殊的znode锁，该锁使用了ZooKeeper对短暂节点的支持，如果会话终止，锁节点将自动删除。
　　　　基于ZooKeeper的选择：
　　　　　　如果本地NameNode是健康的，且ZKFC发现没有其它的节点当前持有znode锁，它将为自己获取该锁。如果成功，则它已经赢得了选择，并负责运行故障转移进程以使它的本地NameNode为active。故障转移进程与前面描述的手动故障转移相似，首先将之前Active NameNode进行降级操作，然后本地NameNode转换为active状态。这样做的好处就是防止脑裂，换句话说，尽管之前的Actice NameNode进程启动成功，它也依旧无法成为Active，而是乖乖的做Standby状态，当另一个Activ节点挂掉时它才会发生状态变化。

三.搭建zookeeper集群

1>.下载zookeeper软件（下载地址：https://www.apache.org/dyn/closer.cgi/zookeeper/）

[root@node101.yinzhengjie.org.cn ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
--2019-04-12 08:48:40--  https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37676320 (36M) [application/x-gzip]
Saving to: ‘zookeeper-3.4.14.tar.gz’

100%[============================================================================================================================================>] 37,676,320  7.57MB/s   in 4.5s   

2019-04-12 08:48:45 (7.94 MB/s) - ‘zookeeper-3.4.14.tar.gz’ saved [37676320/37676320]

[root@node101.yinzhengjie.org.cn ~]# 

2>.加压zookeeper并配置环境变量

[root@node101.yinzhengjie.org.cn ~]# tar -zxf zookeeper-3.4.14.tar.gz -C /yinzhengjie/softwares/
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/zookeeper-3.4.14/
total 1672
drwxr-xr-x  2 2002 2002     202 Mar  6 09:09 bin
-rw-rw-r--  1 2002 2002   97426 Mar  6 08:50 build.xml
drwxr-xr-x  2 2002 2002      77 Mar  6 09:09 conf
drwxr-xr-x  2 2002 2002    4096 Mar  6 09:10 dist-maven
-rw-rw-r--  1 2002 2002    1709 Mar  6 08:50 ivysettings.xml
-rw-rw-r--  1 2002 2002   10742 Mar  6 08:50 ivy.xml
drwxr-xr-x  4 2002 2002     308 Mar  6 09:09 lib
-rw-rw-r--  1 2002 2002   11970 Mar  6 08:50 LICENSE.txt
-rw-rw-r--  1 2002 2002    3132 Mar  6 08:50 NOTICE.txt
-rw-rw-r--  1 2002 2002   31622 Mar  6 08:50 pom.xml
-rw-rw-r--  1 2002 2002    1765 Mar  6 08:50 README.md
-rw-rw-r--  1 2002 2002    1770 Mar  6 08:50 README_packaging.txt
drwxr-xr-x  3 2002 2002      22 Mar  6 08:50 src
-rw-rw-r--  1 2002 2002 1515359 Mar  6 08:50 zookeeper-3.4.14.jar
-rw-rw-r--  1 2002 2002     836 Mar  6 09:10 zookeeper-3.4.14.jar.asc
-rw-rw-r--  1 2002 2002      33 Mar  6 08:50 zookeeper-3.4.14.jar.md5
-rw-rw-r--  1 2002 2002      41 Mar  6 08:50 zookeeper-3.4.14.jar.sha1
drwxr-xr-x  3 2002 2002      47 Mar  6 09:09 zookeeper-client
drwxr-xr-x 12 2002 2002    4096 Mar  6 09:09 zookeeper-contrib
drwxr-xr-x  7 2002 2002    4096 Mar  6 09:09 zookeeper-docs
drwxr-xr-x  3 2002 2002      35 Mar  6 09:09 zookeeper-it
drwxr-xr-x  4 2002 2002      46 Mar  6 09:09 zookeeper-jute
drwxr-xr-x  5 2002 2002     176 Mar  6 09:09 zookeeper-recipes
drwxr-xr-x  3 2002 2002      32 Mar  6 09:09 zookeeper-server
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# tail -3 /etc/profile
#ADD ZOOKEEPER_HOME PATH
ZOOKEEPER_HOME=/yinzhengjie/softwares/zookeeper-3.4.14
PATH=$PATH:$ZOOKEEPER_HOME/bin
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# source /etc/profile
[root@node101.yinzhengjie.org.cn ~]# 

3>.配置zookeeper的配置文件

[root@node101.yinzhengjie.org.cn ~]# cp /yinzhengjie/softwares/zookeeper-3.4.14/conf/zoo_sample.cfg /yinzhengjie/softwares/zookeeper-3.4.14/conf/zoo.cfg 
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/zookeeper-3.4.14/conf/
total 16
-rw-rw-r-- 1 2002 2002  535 Mar  6 08:50 configuration.xsl
-rw-rw-r-- 1 2002 2002 2161 Mar  6 08:50 log4j.properties
-rw-r--r-- 1 root root  922 Apr 12 08:55 zoo.cfg
-rw-rw-r-- 1 2002 2002  922 Mar  6 08:50 zoo_sample.cfg
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/zookeeper-3.4.14/conf/zoo.cfg 
# 滴答，计时的基本单位，默认是2000毫秒，即2秒。它是zookeeper最小的时间单位，用于丈量心跳时间和超时时间等，通常设置成默认2秒即可。
tickTime=2000

# 初始化限制是10滴答，默认是10个滴答，即默认是20秒。指定follower节点初始化是链接leader节点的最大tick次数。
initLimit=5

# 数据同步的时间限制，默认是5个滴答，即默认时间是10秒。设定了follower节点与leader节点进行同步的最大时间。与initLimit类似，它也是以tickTime为单位进行指定的。
syncLimit=2

# 指定zookeeper的工作目录，这是一个非常重要的参数，zookeeper会在内存中在内存只能中保存系统快照，并定期写入该路径指定的文件夹中。生产环境中需要注意该文件夹的磁盘占用情况。
dataDir=/home/yinzhengjie/zookeeper

# 监听zookeeper的默认端口。zookeeper监听客户端链接的端口，一般设置成默认2181即可。
clientPort=2181

# 这个操作将限制连接到 ZooKeeper 的客户端的数量，限制并发连接的数量，它通过 IP 来区分不同的客户端。此配置选项可以用来阻止某些类别的 Dos 攻击。将它设置为 0 或者忽略而不进行设置将会取消对并发连接的限制。
#maxClientCnxns=60
 
# 在上文中已经提到，3.4.0及之后版本，ZK提供了自动清理事务日志和快照文件的功能，这个参数指定了清理频率，单位是小时，需要配置一个1或更大的整数，默认是0，表示不开启自动清理功能。
#autopurge.purgeInterval=1

# 这个参数和上面的参数搭配使用，这个参数指定了需要保留的文件数目。默认是保留3个。
#autopurge.snapRetainCount=3

#server.x=[hostname]:nnnnn[:nnnnn]，这里的x是一个数字，与myid文件中的id是一致的。右边可以配置两个端口，第一个端口用于F和L之间的数据同步和其它通信，第二个端口用于Leader选举过程中投票通信。  
server.101=node101.yinzhengjie.org.cn:2888:3888
server.102=node102.yinzhengjie.org.cn:2888:3888
server.103=node103.yinzhengjie.org.cn:2888:3888
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/zookeeper-3.4.14/conf/java.env 
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com

#指定JDK的安装路径
export JAVA_HOME=/yinzhengjie/softwares/jdk

#指定zookeeper的heap内存大小
export JVMFLAGS="-Xms256m -Xmx256m $JVMFLAGS"
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

4>.编写zookeeper的启动脚本

[root@node101.yinzhengjie.org.cn ~]# vi /usr/local/bin/xzk.sh
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# cat /usr/local/bin/xzk.sh
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com

#判断用户是否传参
if [ $# -ne 1 ];then
    echo "无效参数，用法为: $0  {start|stop|restart|status}"
    exit
fi

#获取用户输入的命令
cmd=$1

#定义函数功能
function zookeeperManger(){
    case $cmd in
    start)
        echo "启动服务"        
        remoteExecution start
        ;;
    stop)
        echo "停止服务"
        remoteExecution stop
        ;;
    restart)
        echo "重启服务"
        remoteExecution restart
        ;;
    status)
        echo "查看状态"
        remoteExecution status
        ;;
    *)
        echo "无效参数，用法为: $0  {start|stop|restart|status}"
        ;;
    esac
}


#定义执行的命令
function remoteExecution(){
    for (( i=101 ; i<=103 ; i++ )) ; do
            tput setaf 2
            echo ========== node${i}.yinzhengjie.org.cn zkServer.sh  $1 ================
            tput setaf 9
            ssh node${i}.yinzhengjie.org.cn  "source /etc/profile ; zkServer.sh $1"
    done
}

#调用函数
zookeeperManger
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# chmod +x /usr/local/bin/xzk.sh
[root@node101.yinzhengjie.org.cn ~]# ll /usr/local/bin/xzk.sh 
-rwxr-xr-x 1 root root 1125 Apr 18 15:54 /usr/local/bin/xzk.sh
[root@node101.yinzhengjie.org.cn ~]# 

5>.创建myid文件并写入服务器编号

[root@node101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir /home/yinzhengjie/zookeeper/'
 [WARNING]: Consider using file module with state=directory rather than running mkdir

node101.yinzhengjie.org.cn | SUCCESS | rc=0 >>


node103.yinzhengjie.org.cn | SUCCESS | rc=0 >>


node102.yinzhengjie.org.cn | SUCCESS | rc=0 >>


[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# for (( i=101;i<=103;i++ )) do ssh node${i}.yinzhengjie.org.cn "echo -n $i > /home/yinzhengjie/zookeeper/myid" ;done
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /home/yinzhengjie/zookeeper/myid'
node102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
102

node103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
103

node101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
101

[root@node101.yinzhengjie.org.cn ~]# 

 

6>.分发配置文件 

[root@node101.yinzhengjie.org.cn ~]# scp /etc/profile node102.yinzhengjie.org.cn:/etc/profile                                                                                                                                             100% 2127     1.2MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# scp /etc/profile node103.yinzhengjie.org.cn:/etc/profile                                                                                                                                             100% 2127     1.4MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/zookeeper-3.4.14/ node102.yinzhengjie.org.cn:/yinzhengjie/softwares/
........
log.42                                                                                                                                              100%  184   128.3KB/s   00:00    
log.63b                                                                                                                                             100%   48KB  22.7MB/s   00:00    
snapshot.0                                                                                                                                          100%  296   277.6KB/s   00:00    
snapshot.272                                                                                                                                        100%   55KB  26.5MB/s   00:00    
snapshot.273                                                                                                                                        100%   55KB  24.3MB/s   00:00    
snapshot.639                                                                                                                                        100%  140KB  27.2MB/s   00:00    
snapshot.83f                                                                                                                                        100% 4824     3.5MB/s   00:00    
minikdc-krb5.conf                                                                                                                                   100% 1103     1.3MB/s   00:00    
minikdc.ldiff                                                                                                                                       100% 1630     2.2MB/s   00:00    
log.100000001                                                                                                                                       100% 2401KB  50.2MB/s   00:00    
log.100001bf0                                                                                                                                       100% 1001KB  59.6MB/s   00:00    
snapshot.100000000                                                                                                                                  100%   73    17.1KB/s   00:00    
snapshot.100001bec                                                                                                                                  100% 2507KB  61.6MB/s   00:00    
check_compatibility.py                                                                                                                              100% 7212     4.7MB/s   00:00    
checkstyle-noframes-sorted.xsl                                                                                                                      100% 5489     6.5MB/s   00:00    
checkstyle.xml                                                                                                                                      100% 7489     7.3MB/s   00:00    
findbugsExcludeFile.xml                                                                                                                             100% 5027     6.6MB/s   00:00    
log4j.properties                                                                                                                                    100% 2712     1.9MB/s   00:00    
test-patch.properties                                                                                                                               100%  855   930.7KB/s   00:00    
test-github-pr.sh                                                                                                                                   100%   24KB   9.7MB/s   00:00    
test-patch.sh                                                                                                                                       100%   24KB  19.4MB/s   00:00    
pom.xml                                                                                                                                             100%   10KB   7.6MB/s   00:00    
LICENSE.txt                                                                                                                                         100%   12KB   5.9MB/s   00:00    
NOTICE.txt                                                                                                                                          100% 3132     1.2MB/s   00:00    
README.md                                                                                                                                           100% 1765     1.4MB/s   00:00    
README_packaging.txt                                                                                                                                100% 1770     2.5MB/s   00:00    
build.xml                                                                                                                                           100%   95KB  39.5MB/s   00:00    
ivy.xml                                                                                                                                             100%   10KB   8.2MB/s   00:00    
ivysettings.xml                                                                                                                                     100% 1709     1.3MB/s   00:00    
pom.xml                                                                                                                                             100%   31KB   8.9MB/s   00:00    
zookeeper-3.4.14.jar                                                                                                                                100% 1480KB  41.5MB/s   00:00    
zookeeper-3.4.14.jar.md5                                                                                                                            100%   33    25.6KB/s   00:00    
zookeeper-3.4.14.jar.sha1                                                                                                                           100%   41    60.7KB/s   00:00    
zookeeper-3.4.14.jar.asc                                                                                                                            100%  836     1.2MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/zookeeper-3.4.14/ node103.yinzhengjie.org.cn:/yinzhengjie/softwares/
........
log.42                                                                                                                                              100%  184   128.3KB/s   00:00    
log.63b                                                                                                                                             100%   48KB  22.7MB/s   00:00    
snapshot.0                                                                                                                                          100%  296   277.6KB/s   00:00    
snapshot.272                                                                                                                                        100%   55KB  26.5MB/s   00:00    
snapshot.273                                                                                                                                        100%   55KB  24.3MB/s   00:00    
snapshot.639                                                                                                                                        100%  140KB  27.2MB/s   00:00    
snapshot.83f                                                                                                                                        100% 4824     3.5MB/s   00:00    
minikdc-krb5.conf                                                                                                                                   100% 1103     1.3MB/s   00:00    
minikdc.ldiff                                                                                                                                       100% 1630     2.2MB/s   00:00    
log.100000001                                                                                                                                       100% 2401KB  50.2MB/s   00:00    
log.100001bf0                                                                                                                                       100% 1001KB  59.6MB/s   00:00    
snapshot.100000000                                                                                                                                  100%   73    17.1KB/s   00:00    
snapshot.100001bec                                                                                                                                  100% 2507KB  61.6MB/s   00:00    
check_compatibility.py                                                                                                                              100% 7212     4.7MB/s   00:00    
checkstyle-noframes-sorted.xsl                                                                                                                      100% 5489     6.5MB/s   00:00    
checkstyle.xml                                                                                                                                      100% 7489     7.3MB/s   00:00    
findbugsExcludeFile.xml                                                                                                                             100% 5027     6.6MB/s   00:00    
log4j.properties                                                                                                                                    100% 2712     1.9MB/s   00:00    
test-patch.properties                                                                                                                               100%  855   930.7KB/s   00:00    
test-github-pr.sh                                                                                                                                   100%   24KB   9.7MB/s   00:00    
test-patch.sh                                                                                                                                       100%   24KB  19.4MB/s   00:00    
pom.xml                                                                                                                                             100%   10KB   7.6MB/s   00:00    
LICENSE.txt                                                                                                                                         100%   12KB   5.9MB/s   00:00    
NOTICE.txt                                                                                                                                          100% 3132     1.2MB/s   00:00    
README.md                                                                                                                                           100% 1765     1.4MB/s   00:00    
README_packaging.txt                                                                                                                                100% 1770     2.5MB/s   00:00    
build.xml                                                                                                                                           100%   95KB  39.5MB/s   00:00    
ivy.xml                                                                                                                                             100%   10KB   8.2MB/s   00:00    
ivysettings.xml                                                                                                                                     100% 1709     1.3MB/s   00:00    
pom.xml                                                                                                                                             100%   31KB   8.9MB/s   00:00    
zookeeper-3.4.14.jar                                                                                                                                100% 1480KB  41.5MB/s   00:00    
zookeeper-3.4.14.jar.md5                                                                                                                            100%   33    25.6KB/s   00:00    
zookeeper-3.4.14.jar.sha1                                                                                                                           100%   41    60.7KB/s   00:00    
zookeeper-3.4.14.jar.asc                                                                                                                            100%  836     1.2MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]# 

7>.启动zookeeper集群

[root@node101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
node101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
Jps

node103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
Jps

node102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
Jps

[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# xzk.sh start
启动服务
========== node101.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== node102.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== node103.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# xzk.sh status
查看状态
========== node101.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: follower
========== node102.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: leader
========== node103.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: follower
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]#  ansible all -m shell -a 'jps'
node103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
QuorumPeerMain
Jps

node102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
QuorumPeerMain
Jps

node101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
QuorumPeerMain
Jps

[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# zkCli.sh 
Connecting to localhost:2181
2019-04-12 09:23:04,270 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2019-04-12 09:23:04,272 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node101.yinzhengjie.org.cn
2019-04-12 09:23:04,273 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-04-12 09:23:04,275 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
2019-04-12 09:23:04,276 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-04-12 09:23:04,276 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-04-12 09:23:04,276 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
2019-04-12 09:23:04,277 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2019-04-12 09:23:04,303 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-12 09:23:04,365 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-04-12 09:23:04,395 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x6500007f14d00000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /zookeeper
[quota]
[zk: localhost:2181(CONNECTED) 2] ls /zookeeper/quota
[]
[zk: localhost:2181(CONNECTED) 3] quit
Quitting...
2019-04-12 09:23:13,101 [myid:] - INFO  [main:ZooKeeper@693] - Session: 0x6500007f14d00000 closed
2019-04-12 09:23:13,102 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@522] - EventThread shut down for session: 0x6500007f14d00000
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

四.配置HDFS HA集群

1>.官网文档（http://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html） 

2>.将集群的完全分布式配置备份一份，再行修改操作

[root@node101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/hadoop-2.9.2/etc/
total 4
drwxr-xr-x. 2 501 dialout 4096 Apr 18 16:03 hadoop
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# cp -r /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/ /yinzhengjie/softwares/hadoop-2.9.2/etc/full
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/hadoop-2.9.2/etc/ 
total 8
drwxr-xr-x 2 root root 4096 Apr 18 16:26 full
drwxr-xr-x. 2 501 dialout 4096 Apr 18 16:03 hadoop
[root@node101.yinzhengjie.org.cn ~]#

3>.配置hadoop-env-sh配置文件

[root@node101.yinzhengjie.org.cn ~]# which java　　　　　　　　　　　　　　　　　　#由于我们实验环境按照的是OpenJDK，默认的按照后的命令被放在了/usr下，因此我们应该在Hadoop的环境变啦配置文件中指定本季的Java命令的路径位置！
/usr/bin/java
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# grep JAVA_HOME /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/hadoop-env.sh | grep -v ^#
export JAVA_HOME=/usr
[root@node101.yinzhengjie.org.cn ~]# 

4>.配置core-site.xml配置文件

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://yinzhengjie-hdfs</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/data/hadoop/ha</value>
        </property>
</configuration>


<!--

core-site.xml配置文件的作用：
    用于定义系统级别的参数，如HDFS URL、Hadoop的临时目录以及用于rack-aware集群中的配置文件的配置等，此中的参数定义会覆盖core-default.xml文件中的默认配置。

fs.defaultFS 参数的作用：
        #声明namenode的地址，相当于声明hdfs文件系统。

hadoop.tmp.dir 参数的作用：
        #声明hadoop工作目录的地址。

-->

[root@node101.yinzhengjie.org.cn ~]# 

5>配置hdfs-site.xml配置文件

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/hdfs-site.xml 
<configuration>
        <!-- 完全分布式集群名称,注意这个名称需要和core-site.xml中定义的fs.defaultFS要一致哟！ -->
        <property>
                <name>dfs.nameservices</name>
                <value>yinzhengjie-hdfs</value>
        </property>

        <!-- 集群中NameNode节点都有哪些 -->
        <property>
                <name>dfs.ha.namenodes.yinzhengjie-hdfs</name>
                <value>namenode1,namenode2</value>
        </property>

        <!-- namenode1的RPC通信地址 -->
        <property>
                <name>dfs.namenode.rpc-address.yinzhengjie-hdfs.namenode1</name>
                <value>node101.yinzhengjie.org.cn:9000</value>
        </property>

        <!-- namenode2的RPC通信地址 -->
        <property>
                <name>dfs.namenode.rpc-address.yinzhengjie-hdfs.namenode2</name>
                <value>node103.yinzhengjie.org.cn:9000</value>
        </property>

        <!-- namenode1的http通信地址 -->
        <property>
                <name>dfs.namenode.http-address.yinzhengjie-hdfs.namenode1</name>
                <value>node101.yinzhengjie.org.cn:50070</value>
        </property>

        <!-- namenode2的http通信地址 -->
        <property>
                <name>dfs.namenode.http-address.yinzhengjie-hdfs.namenode2</name>
                <value>node103.yinzhengjie.org.cn:50070</value>
        </property>

        <!-- 指定NameNode元数据在JournalNode上的存放位置 -->
        <property>
                <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://node103.yinzhengjie.org.cn:8485;node102.yinzhengjie.org.cn:8485;node101.yinzhengjie.org.cn:8485/yinzhengjie-hdfs</value>
        </property>

        <!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
        <property>
                <name>dfs.ha.fencing.methods</name>
                <value>sshfence</value>
        </property>

        <!-- 使用隔离机制时需要ssh无秘钥登录-->
        <property>
                <name>dfs.ha.fencing.ssh.private-key-files</name>
                <value>/root/.ssh/id_rsa</value>
        </property>

        <!-- 声明journalnode服务器存储目录-->
        <property>
                <name>dfs.journalnode.edits.dir</name>
                <value>/data/hadoop/jn</value>
        </property>

        <!-- 关闭权限检查-->
        <property>
                <name>dfs.permissions.enable</name>
                <value>false</value>
        </property>

        <property>
                <name>dfs.ha.automatic-failover.enabled</name>
                <value>true</value>
        </property>


        <property>
                <name>ha.zookeeper.quorum</name>
                <value>node101.yinzhengjie.org.cn:2181,node102.yinzhengjie.org.cn:2181,node103.yinzhengjie.org.cn:2181</value>
        </property>
        <!-- 访问代理类：client，mycluster，active配置失败自动切换实现方式-->
        <property>
                <name>dfs.client.failover.proxy.provider.yinzhengjie-hdfs</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
        </property>
</configuration>
[root@node101.yinzhengjie.org.cn ~]# 

6>.拷贝配置好的Hadoop环境到其他节点

[root@node101.yinzhengjie.org.cn ~]# scp  -r /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/ node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/
mapred-site.xml.template                                                                                                                                                                                                  100%  758   423.1KB/s   00:00    
hadoop-metrics2.properties                                                                                                                                                                                                100% 2598     2.0MB/s   00:00    
ssl-server.xml.example                                                                                                                                                                                                    100% 2697     4.0MB/s   00:00    
mapred-site.xml                                                                                                                                                                                                           100%  769   822.6KB/s   00:00    
slaves                                                                                                                                                                                                                    100%   81   104.5KB/s   00:00    
yarn-site.xml                                                                                                                                                                                                             100%  693     1.3MB/s   00:00    
hadoop-env.sh-2019-04-11                                                                                                                                                                                                  100% 4969     5.5MB/s   00:00    
yarn-env.cmd                                                                                                                                                                                                              100% 2250     4.3MB/s   00:00    
ssl-client.xml.example                                                                                                                                                                                                    100% 2316     4.1MB/s   00:00    
httpfs-env.sh                                                                                                                                                                                                             100% 2230     4.7MB/s   00:00    
kms-acls.xml                                                                                                                                                                                                              100% 3518     6.2MB/s   00:00    
httpfs-signature.secret                                                                                                                                                                                                   100%   21    41.3KB/s   00:00    
capacity-scheduler.xml                                                                                                                                                                                                    100% 7861    12.6MB/s   00:00    
yarn-env.sh                                                                                                                                                                                                               100% 4876    10.3MB/s   00:00    
mapred-env.sh                                                                                                                                                                                                             100% 1507     3.0MB/s   00:00    
httpfs-log4j.properties                                                                                                                                                                                                   100% 1657     2.7MB/s   00:00    
kms-env.sh                                                                                                                                                                                                                100% 3139     4.2MB/s   00:00    
kms-site.xml                                                                                                                                                                                                              100% 5939     7.1MB/s   00:00    
hadoop-env.cmd                                                                                                                                                                                                            100% 4133     7.8MB/s   00:00    
container-executor.cfg                                                                                                                                                                                                    100% 1211     2.3MB/s   00:00    
mapred-queues.xml.template                                                                                                                                                                                                100% 4113     7.1MB/s   00:00    
kms-log4j.properties                                                                                                                                                                                                      100% 1788     3.3MB/s   00:00    
hadoop-policy.xml                                                                                                                                                                                                         100%   10KB  12.1MB/s   00:00    
hdfs-site.xml                                                                                                                                                                                                             100% 2283     3.9MB/s   00:00    
httpfs-site.xml                                                                                                                                                                                                           100%  620     1.2MB/s   00:00    
core-site.xml                                                                                                                                                                                                             100%  761     1.4MB/s   00:00    
hadoop-metrics.properties                                                                                                                                                                                                 100% 2490     3.5MB/s   00:00    
hadoop-env.sh                                                                                                                                                                                                             100% 4961     8.7MB/s   00:00    
mapred-env.cmd                                                                                                                                                                                                            100% 1076     1.9MB/s   00:00    
configuration.xsl                                                                                                                                                                                                         100% 1335     2.4MB/s   00:00    
log4j.properties                                                                                                                                                                                                          100%   14KB  17.0MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# scp  -r /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/ node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/
mapred-site.xml.template                                                                                                                                                                                                  100%  758   639.0KB/s   00:00    
hadoop-metrics2.properties                                                                                                                                                                                                100% 2598     2.9MB/s   00:00    
ssl-server.xml.example                                                                                                                                                                                                    100% 2697     4.1MB/s   00:00    
mapred-site.xml                                                                                                                                                                                                           100%  769     2.4MB/s   00:00    
slaves                                                                                                                                                                                                                    100%   81   119.6KB/s   00:00    
yarn-site.xml                                                                                                                                                                                                             100%  693     1.7MB/s   00:00    
hadoop-env.sh-2019-04-11                                                                                                                                                                                                  100% 4969     4.5MB/s   00:00    
yarn-env.cmd                                                                                                                                                                                                              100% 2250     2.9MB/s   00:00    
ssl-client.xml.example                                                                                                                                                                                                    100% 2316     3.1MB/s   00:00    
httpfs-env.sh                                                                                                                                                                                                             100% 2230     4.2MB/s   00:00    
kms-acls.xml                                                                                                                                                                                                              100% 3518     6.5MB/s   00:00    
httpfs-signature.secret                                                                                                                                                                                                   100%   21    42.6KB/s   00:00    
capacity-scheduler.xml                                                                                                                                                                                                    100% 7861     9.3MB/s   00:00    
yarn-env.sh                                                                                                                                                                                                               100% 4876     9.1MB/s   00:00    
mapred-env.sh                                                                                                                                                                                                             100% 1507     3.3MB/s   00:00    
httpfs-log4j.properties                                                                                                                                                                                                   100% 1657     3.4MB/s   00:00    
kms-env.sh                                                                                                                                                                                                                100% 3139     4.7MB/s   00:00    
kms-site.xml                                                                                                                                                                                                              100% 5939    10.5MB/s   00:00    
hadoop-env.cmd                                                                                                                                                                                                            100% 4133     6.5MB/s   00:00    
container-executor.cfg                                                                                                                                                                                                    100% 1211     2.4MB/s   00:00    
mapred-queues.xml.template                                                                                                                                                                                                100% 4113     8.7MB/s   00:00    
kms-log4j.properties                                                                                                                                                                                                      100% 1788     2.8MB/s   00:00    
hadoop-policy.xml                                                                                                                                                                                                         100%   10KB  13.1MB/s   00:00    
hdfs-site.xml                                                                                                                                                                                                             100% 2283     4.8MB/s   00:00    
httpfs-site.xml                                                                                                                                                                                                           100%  620     1.4MB/s   00:00    
core-site.xml                                                                                                                                                                                                             100%  761     2.1MB/s   00:00    
hadoop-metrics.properties                                                                                                                                                                                                 100% 2490     5.2MB/s   00:00    
hadoop-env.sh                                                                                                                                                                                                             100% 4961    11.7MB/s   00:00    
mapred-env.cmd                                                                                                                                                                                                            100% 1076     2.2MB/s   00:00    
configuration.xsl                                                                                                                                                                                                         100% 1335     2.8MB/s   00:00    
log4j.properties                                                                                                                                                                                                          100%   14KB  17.6MB/s   00:00    
[root@node101.yinzhengjie.org.cn ~]#

7>.在各个JournalNode节点上，启动Journalnode服务

[root@node101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start journalnode
starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# jps
15153 Jps
14983 JournalNode
7629 QuorumPeerMain
[root@node101.yinzhengjie.org.cn ~]# 

[root@node102.yinzhengjie.org.cn ~]#  hadoop-daemon.sh start journalnode
starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node102.yinzhengjie.org.cn.out
[root@node102.yinzhengjie.org.cn ~]# 
[root@node102.yinzhengjie.org.cn ~]# jps
11949 QuorumPeerMain
19166 JournalNode
19247 Jps
[root@node102.yinzhengjie.org.cn ~]# 
[root@node102.yinzhengjie.org.cn ~]# 

[root@node103.yinzhengjie.org.cn ~]#  hadoop-daemon.sh start journalnode
starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node103.yinzhengjie.org.cn.out
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# jps
5040 JournalNode
30132 QuorumPeerMain
5146 Jps
[root@node103.yinzhengjie.org.cn ~]# 

8>.在任意一个NameNode节点上进行格式化并启动

[root@node101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# jps
17559 JournalNode
18649 NameNode
18747 Jps
7629 QuorumPeerMain
[root@node101.yinzhengjie.org.cn ~]# 

9>.在另外一个NameNode节点上同步上一步NameNode的元数据信息并启动 

[root@node103.yinzhengjie.org.cn ~]# hdfs namenode -bootstrapStandby
19/04/18 17:11:23 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node103.yinzhengjie.org.cn/172.30.1.103
STARTUP_MSG:   args = [-bootstrapStandby]
STARTUP_MSG:   version = 2.9.2
STARTUP_MSG:   classpath = /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-auth-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/fst-2.50.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-registry-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-router-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-api-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 826afbeae31ca687bc2f8471dc841b66ed2c6704; compiled by 'ajisaka' on 2018-11-13T12:42Z
STARTUP_MSG:   java = 1.8.0_201
************************************************************/
19/04/18 17:11:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
19/04/18 17:11:23 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
=====================================================
About to bootstrap Standby ID namenode2 from:
           Nameservice ID: yinzhengjie-hdfs
        Other Namenode ID: namenode1
  Other NN's HTTP address: http://node101.yinzhengjie.org.cn:50070
  Other NN's IPC  address: node101.yinzhengjie.org.cn/172.30.1.101:9000
             Namespace ID: 688234594
            Block pool ID: BP-1646329568-172.30.1.101-1555578389072
               Cluster ID: CID-eb844a86-4ab9-4a0b-94f8-e8627f99d958
           Layout version: -63
       isUpgradeFinalized: true
=====================================================
19/04/18 17:11:24 INFO common.Storage: Storage directory /data/hadoop/ha/dfs/name has been successfully formatted.
19/04/18 17:11:24 INFO namenode.FSEditLog: Edit logging is async:true
19/04/18 17:11:24 INFO namenode.TransferFsImage: Opening connection to http://node101.yinzhengjie.org.cn:50070/imagetransfer?getimage=1&txid=0&storageInfo=-63:688234594:1555578389072:CID-eb844a86-4ab9-4a0b-94f8-e8627f99d958&bootstrapstandby=true
19/04/18 17:11:24 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
19/04/18 17:11:24 INFO namenode.TransferFsImage: Combined time for fsimage download and fsync to all disks took 0.00s. The fsimage download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /data/hadoop/ha/dfs/name/current/fsimage.ckpt_0000000000000000000 took 0.00s.
19/04/18 17:11:24 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 323 bytes.
19/04/18 17:11:24 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node103.yinzhengjie.org.cn/172.30.1.103
************************************************************/
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# 

[root@node103.yinzhengjie.org.cn ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node103.yinzhengjie.org.cn.out
[root@node103.yinzhengjie.org.cn ~]# jps
7506 JournalNode
30132 QuorumPeerMain
9544 Jps
9452 NameNode
[root@node103.yinzhengjie.org.cn ~]# 

10>.启动所有的datanode

[root@node101.yinzhengjie.org.cn ~]# jps
17559 JournalNode
18649 NameNode
26219 Jps
7629 QuorumPeerMain
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start datanode
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# jps
17559 JournalNode
18649 NameNode
26393 Jps
7629 QuorumPeerMain
26286 DataNode
[root@node101.yinzhengjie.org.cn ~]# 

[root@node102.yinzhengjie.org.cn ~]# jps
11949 QuorumPeerMain
29631 Jps
21471 JournalNode
[root@node102.yinzhengjie.org.cn ~]# hadoop-daemon.sh start datanode
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node102.yinzhengjie.org.cn.out
[root@node102.yinzhengjie.org.cn ~]# 
[root@node102.yinzhengjie.org.cn ~]# jps
11949 QuorumPeerMain
29677 DataNode
29791 Jps
21471 JournalNode
[root@node102.yinzhengjie.org.cn ~]# 

[root@node103.yinzhengjie.org.cn ~]# jps
7506 JournalNode
30132 QuorumPeerMain
16183 Jps
9452 NameNode
[root@node103.yinzhengjie.org.cn ~]# hadoop-daemon.sh start datanode
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node103.yinzhengjie.org.cn.out
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# jps
16224 DataNode
7506 JournalNode
30132 QuorumPeerMain
9452 NameNode
16332 Jps
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# 

11>.手动将node101.yinzhengjie.org.cn节点切换为Active状态

[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1    
standby
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -transitionToActive namenode1 --forcemanual　　　　　　　　　　#我们可以手动强制切换！但会他会弹出提示信息，说是手动切换可能会导致脑裂的发生！
You have specified the --forcemanual flag. This flag is dangerous, as it can induce a split-brain scenario that WILL CORRUPT your HDFS namespace, possibly irrecoverably.

It is recommended not to use this flag, but instead to shut down the cluster and disable automatic failover if you prefer to manually manage your HA state.

You may abort safely by answering 'n' or hitting ^C now.

Are you sure you want to continue? (Y or N) y　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#问我们是否迹象，我们输入y即可！
19/04/18 17:50:29 WARN ha.HAAdmin: Proceeding with manual HA state management even though
automatic failover is enabled for NameNode at node103.yinzhengjie.org.cn/172.30.1.103:9000
19/04/18 17:50:29 WARN ha.HAAdmin: Proceeding with manual HA state management even though
automatic failover is enabled for NameNode at node101.yinzhengjie.org.cn/172.30.1.101:9000
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1                 
active
[root@node101.yinzhengjie.org.cn ~]# 

12>.配置HDFS HA自动故障转移

[root@node101.yinzhengjie.org.cn ~]# stop-dfs.sh 
Stopping namenodes on [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node101.yinzhengjie.org.cn: stopping namenode
node103.yinzhengjie.org.cn: stopping namenode
node103.yinzhengjie.org.cn: stopping datanode
node101.yinzhengjie.org.cn: stopping datanode
node102.yinzhengjie.org.cn: stopping datanode
Stopping journal nodes [node103.yinzhengjie.org.cn node102.yinzhengjie.org.cn node101.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: stopping journalnode
node102.yinzhengjie.org.cn: stopping journalnode
node101.yinzhengjie.org.cn: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: no zkfc to stop
node101.yinzhengjie.org.cn: no zkfc to stop
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# xzk.sh start
启动服务
========== node101.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== node102.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== node103.yinzhengjie.org.cn zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# xzk.sh status
查看状态
========== node101.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: leader
========== node102.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: follower
========== node103.yinzhengjie.org.cn zkServer.sh status ================
ZooKeeper JMX enabled by default
Using config: /yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: follower
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# hdfs zkfc -formatZK
19/04/18 18:01:30 INFO tools.DFSZKFailoverController: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG:   host = node101.yinzhengjie.org.cn/172.30.1.101
STARTUP_MSG:   args = [-formatZK]
STARTUP_MSG:   version = 2.9.2
STARTUP_MSG:   classpath = /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-auth-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/fst-2.50.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-api-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-router-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-registry-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 826afbeae31ca687bc2f8471dc841b66ed2c6704; compiled by 'ajisaka' on 2018-11-13T12:42Z
STARTUP_MSG:   java = 1.8.0_201
************************************************************/
19/04/18 18:01:30 INFO tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
19/04/18 18:01:30 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at node101.yinzhengjie.org.cn/172.30.1.101:9000
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:host.name=node101.yinzhengjie.org.cn
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_201
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hadoop-auth-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/hdfs/hadoop-hdfs-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-configuration-1.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/xmlenc-0.52.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-framework-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-smart-1.3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-net-3.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-digester-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/activation-1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/fst-2.50.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-api-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-router-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-client-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-registry-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar:/yinzhengjie/softwares/hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/contrib/capacity-scheduler/*.jar
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/yinzhengjie/softwares/hadoop-2.9.2/lib/native
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-957.el7.x86_64
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:user.name=root
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node101.yinzhengjie.org.cn:2181,node102.yinzhengjie.org.cn:2181,node103.yinzhengjie.org.cn:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@654f0d9c
19/04/18 18:01:30 INFO zookeeper.ClientCnxn: Opening socket connection to server node102.yinzhengjie.org.cn/172.30.1.102:2181. Will not attempt to authenticate using SASL (unknown error)
19/04/18 18:01:30 INFO zookeeper.ClientCnxn: Socket connection established to node102.yinzhengjie.org.cn/172.30.1.102:2181, initiating session
19/04/18 18:01:30 INFO zookeeper.ClientCnxn: Session establishment complete on server node102.yinzhengjie.org.cn/172.30.1.102:2181, sessionid = 0x6600007830b30000, negotiated timeout = 10000
19/04/18 18:01:30 INFO ha.ActiveStandbyElector: Session connected.
19/04/18 18:01:30 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/yinzhengjie-hdfs in ZK.
19/04/18 18:01:30 INFO zookeeper.ZooKeeper: Session: 0x6600007830b30000 closed
19/04/18 18:01:30 INFO zookeeper.ClientCnxn: EventThread shut down
19/04/18 18:01:30 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at node101.yinzhengjie.org.cn/172.30.1.101
************************************************************/
[root@node101.yinzhengjie.org.cn ~]# 

[root@node101.yinzhengjie.org.cn ~]# jps
31104 Jps
29947 QuorumPeerMain
30718 ZooKeeperMain
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh 
Starting namenodes on [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node103.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node103.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node102.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node101.yinzhengjie.org.cn.out
Starting journal nodes [node103.yinzhengjie.org.cn node102.yinzhengjie.org.cn node101.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node103.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node101.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node102.yinzhengjie.org.cn.out
Starting ZK Failover Controllers on NN hosts [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-zkfc-node103.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-zkfc-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# jps
31376 DataNode
31600 JournalNode
29947 QuorumPeerMain
31246 NameNode
30718 ZooKeeperMain
31902 Jps
31807 DFSZKFailoverController
[root@node101.yinzhengjie.org.cn ~]# 

五.验证HDFS HA的可用性（注意，如果你执行下面的操作之前，请确认你两个NameNode之间是否配置了免密码登陆！否则可能看不到我博客中展示的效果哟！）

1>.将Active NameNode进程杀死，观察Standby NameNode是否会自动切换

[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1　　　　　　　　　　　　　　　　　　#查看当前节点的状态
active
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# kill -9 `jps | grep NameNode | awk '{print $1}' `　　　　　　　　　　　　#直接杀掉NameNode进程
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1
19/04/18 18:15:24 INFO ipc.Client: Retrying connect to server: node101.yinzhengjie.org.cn/172.30.1.101:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From node101.yinzhengjie.org.cn/172.30.1.101 to node101.yinzhengjie.org.cn:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
[root@node101.yinzhengjie.org.cn ~]# 

[zk: localhost:2181(CONNECTED) 0] ls /hadoop-ha/yinzhengjie-hdfs
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: localhost:2181(CONNECTED) 1] 

[zk: localhost:2181(CONNECTED) 1] get /hadoop-ha/yinzhengjie-hdfs/ActiveStandbyElectorLock

yinzhengjie-hdfs        namenode2node103.yinzhengjie.org.cn �F(�>
cZxid = 0x2000001fc
ctime = Thu Apr 18 18:34:24 CST 2019
mZxid = 0x2000001fc
mtime = Thu Apr 18 18:34:24 CST 2019
pZxid = 0x2000001fc
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x65001122c5ce0036
dataLength = 63
numChildren = 0
[zk: localhost:2181(CONNECTED) 2] 

[zk: localhost:2181(CONNECTED) 2] get /hadoop-ha/yinzhengjie-hdfs/ActiveBreadCrumb

yinzhengjie-hdfs        namenode2node103.yinzhengjie.org.cn �F(�>
cZxid = 0x200000009
ctime = Thu Apr 18 18:02:56 CST 2019
mZxid = 0x2000001fd
mtime = Thu Apr 18 18:34:25 CST 2019
pZxid = 0x200000009
cversion = 0
dataVersion = 3
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 63
numChildren = 0
[zk: localhost:2181(CONNECTED) 3] 

2>.将Active NameNode机器网络断开，观察Standby NameNode是否会自动切换

[root@node101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start namenode　　　　　　　　　　　　　　　　　　　　 #首先我们把刚刚杀死的NamenNode进程启动起来。
starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1　　　　　　　　　　　　　　　　#启动后，发现其状态为standby。
standby
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ssh node103.yinzhengjie.org.cn　　　　　　　　　　　　　　　　　　　　　　#远程到active的服务器上，我们要搞点事情。
Last login: Thu Apr 18 16:52:18 2019 from 172.30.1.2
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode2
active
[root@node103.yinzhengjie.org.cn ~]# 
[root@node103.yinzhengjie.org.cn ~]# reboot 　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#我们把服务器重启一下，模拟Active节点的主机故障！
Connection to node103.yinzhengjie.org.cn closed by remote host.
Connection to node103.yinzhengjie.org.cn closed.
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1　　　　　　　　　　　　　　　　　　#再次查看，发现刚刚的Standby状态转换成Active啦！说明切换成功！
active
[root@node101.yinzhengjie.org.cn ~]# 

[zk: localhost:2181(CONNECTED) 11] get /hadoop-ha/yinzhengjie-hdfs/ActiveStandbyElectorLock

yinzhengjie-hdfs        namenode1node101.yinzhengjie.org.cn �F(�>
cZxid = 0x2000001f7
ctime = Thu Apr 18 18:23:13 CST 2019
mZxid = 0x2000001f7
mtime = Thu Apr 18 18:23:13 CST 2019
pZxid = 0x2000001f7
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x6700112119110038
dataLength = 63
numChildren = 0
[zk: localhost:2181(CONNECTED) 12] 

[zk: localhost:2181(CONNECTED) 12] get /hadoop-ha/yinzhengjie-hdfs/ActiveBreadCrumb        

yinzhengjie-hdfs        namenode1node101.yinzhengjie.org.cn �F(�>
cZxid = 0x200000009
ctime = Thu Apr 18 18:02:56 CST 2019
mZxid = 0x2000001f8
mtime = Thu Apr 18 18:23:30 CST 2019
pZxid = 0x200000009
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 63
numChildren = 0
[zk: localhost:2181(CONNECTED) 13]