032 HDFS中高可用性HA的讲解

HDFS Using QJM

　　HA使用的是分布式的日志管理方式

一：概述

1.背景

　　如果namenode出现问题，整个HDFS集群将不能使用。

　　是不是可以有两个namenode呢

　　　　一个为对外服务->active

　　　　一个处于待机->standby

　　　　他们的之间共享的元数据叫 nameservice

2.HDFS HA的几大中重点

　　1）保证两个namenode里面的内存中存储的文件的元数据同步

　　　　->namenode启动时，会读镜像文件

　　2）变化的记录信息同步

　　3）日志文件的安全性

　　　　->分布式的存储日志文件（cloudera公司提出来的）

　　　　　　->2n+1个，使用副本数保证安全性

　　　　->使用zookeeper监控

　　　　　　->监控两个namenode，当一个出现了问题，可以达到自动故障转移。

　　　　　　->如果出现了问题，不会影响整个集群

　　　　　　->zookeeper对时间同步要求比较高。

　　4）客户端如何知道访问哪一个namenode

　　　　->使用proxy代理

　　　　->隔离机制

　　　　->使用的是sshfence

　　　　->两个namenode之间无密码登录

　　5）namenode是哪一个是active

　　　　->zookeeper通过选举选出zookeeper。

　　　　->然后zookeeper开始监控，如果出现文件，自动故障转移。

3.架构

　　注意两个namenode之间的话。

　　

4.使用QJM

　　JournalNode：日志节点，进程管理。

　　

二：准备

3.规划集群

　　namenode　　　　namenode

　　journalnode　　   journalnode　　　　journalnode -->日志的分布，这是日志节点，考虑的是日志的安全性。

　　datanode　　　　 datanode　　　　　 datanode

4.关闭所有的进程

　　

5.保存分布式的源数据，复制一份用来HDFS HA的检测。

　　先是第一台，先将分布式的etc/hadoop，保存为dist-hadoop，保存源数据。

　　同时，新建tmp。

　　至于第二台以及第三台，在分发之间再进行配置。

　　

三：配置文件

6.将core-site.xml中的文件系统删除，并添加新的文件系统

　　以前的是使用是配置一台，现在配置多态namenode，使用的方式是nameservices的名称的方式。

　　

　　添加配置

　　

7.配置hdfs-site.xml

　　

8.继续配置hdfs-site.xml

　　dfs.nameservices的配置

　　

　　dfs.ha.namenodes.[nameservice ID]的配置

　　包括rpc，http的namenodde地址。

　　

　　dfs.namenode.shared.edits.dir的配置

　　这是journalnode的地址

　　

　　dfs.journalnode.edits.dir 的配置

　　这是journalnode的日志存储的目录

　　先新建目录：

　　

　　

　　dfs.client.failover.proxy.provider的配置（客户端）

　　

　　dfs.ha.fencing.methods的配置

　　使用的方式为ssh拦截

　　

　　

　　

9.配置完成，在分发之前先进行的是目录的规划

　　

　　

10.分发

　　

四：启动

1.启动三台zookeeper

　　现在有些奇怪，这三台zookeeper为啥要启动，好像没用到。

　　看视频上启动的，在这里暂时不启动了，因为没有用到，在自动故障转移的时候才去使用。

2.启动三台的日志节点

　　

　　

3.格式化第一台虚拟机

　　因为是共享数据，所以格式化一台虚拟机即可。

　　因为管理的是同一个元数据，所以另外一个不需要格式化。

　　

4.启动namenode（第一台）

5.紧接着，同步元数据（在第二台上写命令）

　　最好是bin/hdfs namenode -help查看

　　

6.启动namenode（第两台虚拟机）

　　

7.启动三台的datanode

　　

8.观看两台的启动状态

　　关于这个问题，刚才在官网上看了一下，的确是启动后会是两个standby。

　　官网如下：

　　

　　

　　

9.强制切换状态

　　1）、查找帮助命令，属于bin/hdfs haadmin

　　

　　2）、具体命令

　　

10.结果

　　1）、

　　

　　或者：

　　这个查看第一台的状态，也可以在第二台虚拟机上查看节点一，例如：

　　

　　2）、

　　

五：再次测试

1.在HDFS上新建目录并上传文件

　　

2.杀死第一台的namenode，进行测试

　　

3.将avtive的状态切换到第二台

　　这个命令应该在第二台虚拟机上进行操作。

　　通过实验发现，如果不进行强制转换成active，这个命令是不会成功的，所以需要添加上--forceactive。

　　

4.看第一台是否可以观看hdfs中的文件

　　如果可以，说明，HA发挥作用了。

　　因为这时proxy提供的接口变成nn2.

　　

六：自动故障转移

　　前提：关闭所有的进程。

　　依赖：zookeeper的监控，组件为：ZKFC。

　　启动以后都是standby，选举一个active。

　　规划：

　　namenode　　　　namenode

　　ZKFC     　　     　ZKFC

　　journalnode　　   journalnode　　　　journalnode

　　datanode　　　　 datanode　　　　　 datanode

　　

1.看看官网的说法

　　这个地方关于zookeeper的使用不是太理解，需要到官网上研究一下。

　　还配置了一个zookeeper集群，用于ZKFC（DFSZKFailoverCont roller）故障转移。

　　文章最后做一下说明。对ZKFC的原理。

　　

2.配置core-site.xml

　　添加zookeeper的服务，包括主机名和端口号。

　　

3.配置hdfs-site.xml

　　添加自动故障转移的使能。

　　

4.分发

　　

5.确定关闭所有的进程

 　　这一步是开始的基础。

　　官网上的说法：

　　Before you begin configuring automatic failover, you should shut down your cluster. It is not currently possible to transition from a manual failover setup to an automatic failover setup while the cluster is running.

6.开启三台的zookeeper服务

　　先开启监控，然后方便下面进行初始化ZK，产生一个ZNode目录。

 　　

7.初始化HA在zookeeper中的状态bin/hdfs zkfc -formatZK

　　在zookepper上创建znode节点，用于存储数据，方便自动实现故障转移。

　　关于这个ZKFC的初始化需要在哪台虚拟机上进行，通过思考与实践，在可以进行HA的那台机器上即可，而且做一次就行了。

　　在下面有说明情况。

　　

8.观察成功与否

　　进入zookeeper目录

　　命令：bin/zkCli.sh

　　

9.启动sbin/start-dfs.sh

　　前两台虚拟机会出现DFZKFailoverController。

　　如果没有开启DFZKFailoverController，可以手动开启，命令是sbin/hadoop-daemon.sh start zkfc。

　　查看官网的说法：

　　

　　

　

七：简单检测

1.展示前两台的虚拟机状态

　　

　　

2.杀死第二台的虚拟机

　　

3.结果

　　这时，第一台虚拟机变成active。

 　　

八：HA流程

1.说明

　　ZKGZ通过RPC协议获取namenode的状态

　　ZKFC连接zookeeper，完成选举。

　　stabdby节点ZKFC监控本机的namenode。

　　ZKFC在监控周期内检测到失效的namenode。

　　ZKFC记录失败的active namenode信息到zookeeper。

　　zookeeper发送一个通知给stabdby的节点ZKFC

　　standby节点ZKFC进入切换的流程，隔离刚才的主机。

　　ZKFC进行选举，确保新的active节点。

九：ZKFC原理说明

1.基本原理

　　zk的基本特性：
　　(1) 可靠存储小量数据且提供强一致性
　　(2) ephemeral node, 在创建它的客户端关闭后，可以自动删除
　　(3) 对于node状态的变化，可以提供异步的通知(watcher)

　　zk在zkfc中可以提供的功能：
　　(1) Failure detector: 及时发现出故障的NN，并通知zkfc
　　(2) Active node locator: 帮助客户端定位哪个是Active的NN
　　(3) Mutual exclusion of active state: 保证某一时刻只有一个Active的NN

2.模块

　　(1) ZKFailoverController(DFSZKFailoverController): 驱动整个ZKFC的运转，通过向HealthMonitor和ActiveStandbyElector注册回调函数的方式，subscribe HealthMonitor和ActiveStandbyElector的事件，并做相应的处理
　　(2) HealthMonitor: 定期check NN的健康状况，在NN健康状况发生变化时，通过回调函数把变化通知给ZKFailoverController
　　(3) ActiveStandbyElector: 管理NN在zookeeper上的状态，zookeeper上对应node的结点发生变化时，通过回调函数把变化通知给ZKFailoverController
　　(4) FailoverController: 提供做graceful failover的相关功能(dfs admin可以通过命令行工具手工发起failover) 

3.架构

　　

4.线程模型

　　ZKFC的线程模型总体上来讲比较简单的，它主要包括三类线程，一是主线程；一是HealthMonitor线程; 一是zookeeper客户端的线程。它们的主要工作方式是：
　　(1) 主线程在启动所有的服务后就开始循环等待
　　(2) HealthMonitor是一个单独的线程，它定期向NN发包，检查NN的健康状况
　　(3) 当NN的状态发生变化时，HealthMonitor线程会回调ZKFailoverController注册进来的回调函数，通知ZKFailoverController NN的状态发生了变化
　　(4) ZKFailoverController收到通知后，会调用ActiveStandbyElector的API，来管理在zookeeper上的结点的状态
　　(5) ActiveStandbyElector会调用zookeeper客户端API监控zookeeper上结点的状态，发生变化时，回调ZKFailoverController的回调函数，通知ZKFailoverController，做出相应的变化

5.类关系图

　　ZKFC的主类是org.apache.hadoop.hdfs.tools.DFSZKFailoverController。

　　

　　

6.formatZK

　　创建特定目录，作为后续写节点状态的父路径。如果该目录已经存在，清理原有目录为空目录。

7.继续说明

•  HealthMonitor

　　在一个独立线程中，通过RPC方式，周期性的调用HAServiceProtocol接口的monitorHealth方法，获取NN的状态。并把状态报告给ActiveStandbyElector 。

•  ActiveStandbyElector

　　ActiveStandbyElector 负责判断哪个NN可以成为Active。它通过ZK，看哪个能够成功的创建一个特定的ephemeral lock file (znode)，哪个就是Active，其它的成为Standby。在一个节点被通知变成Active后，它必须确保自己能够提供一致性的服务（数据一致性），否则它需要主动退出选举。

　　如果一个Active因HealthMonitor监控到状态异常，这里会作出判断，先通过Fenceing功能关闭它（确保关闭或者不能提供服务），然后在ZK上删除它对应ZNode。

　　发送上述事件后，在另外一台机器上的ZKFC中的ActiveStandbyElector 会收到事件，并重新进行选举（尝试创建特定ZNode），它将获得成功并更改NN中状态，从而实现Active节点的变更。

十：formatZK日志说明

1.日志

18/01/28 19:48:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/01/28 19:48:30 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at linux-hadoop01.ibeifeng.com/192.168.140.121:8020
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:host.name=linux-hadoop01.ibeifeng.com
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/modules/jdk1.7.0_67/jre
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/fullyDistributions/ha/hadoop-2.5.0/etc/hadoop:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/contrib/capacity-scheduler/*.jar
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/fullyDistributions/ha/hadoop-2.5.0/lib/native
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.el6.x86_64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.name=beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/fullyDistributions/ha/hadoop-2.5.0
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=linux-hadoop01.ibeifeng.com:2181,linux-hadoop02.ibeifeng.com:2181,linux-hadoop03.ibeifeng.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@32dd803
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Opening socket connection to server linux-hadoop02.ibeifeng.com/192.168.140.122:2181. Will not attempt to authenticate using SASL (unknown error)
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Socket connection established to linux-hadoop02.ibeifeng.com/192.168.140.122:2181, initiating session
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Session establishment complete on server linux-hadoop02.ibeifeng.com/192.168.140.122:2181, sessionid = 0x2613c8b97f50000, negotiated timeout = 5000
18/01/28 19:48:31 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Session: 0x2613c8b97f50000 closed
18/01/28 19:48:31 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2613c8b97f50000
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: EventThread shut down

2.作用

　　在倒数第四行有一行说法。

　　创建了一个目录：

 /hadoop-ha/ns1 in ZK

3.操作

　　上面有日志说connect 02虚拟机。

　　进入第二台zookeeper看看是否存在这个目录。