在docker里面安装hadoop HA 在启动或者停止的时候报ssh不能解析问题。
问题现象:
发现图片不清晰:把问题现象粘贴如下:
root@master:/usr/local/hadoop-2.9.1# start-dfs.sh
Starting namenodes on [2018-10-15 09:45:30,412 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
master slave1]
[main]: ssh: Could not resolve hostname [main]: Name or service not known
(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (nativecodeloader.java:<clinit>(62)): Name or service not known
WARN: ssh: Could not resolve hostname warn: Name or service not known
-: ssh: Could not resolve hostname -: Name or service not known
Unable: ssh: Could not resolve hostname unable: Name or service not known
load: ssh: Could not resolve hostname load: Name or service not known
native-hadoop: ssh: Could not resolve hostname native-hadoop: Name or service not known
for: ssh: Could not resolve hostname for: Name or service not known
2018-10-15: ssh: Could not resolve hostname 2018-10-15: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
platform...: ssh: Could not resolve hostname platform...: Name or service not known
your: ssh: Could not resolve hostname your: Name or service not known
using: ssh: Could not resolve hostname using: Name or service not known
where: ssh: Could not resolve hostname where: Name or service not known
builtin-java: ssh: Could not resolve hostname builtin-java: Name or service not known
util.NativeCodeLoader: ssh: Could not resolve hostname util.nativecodeloader: Name or service not known
applicable: ssh: Could not resolve hostname applicable: Name or service not known
classes: ssh: Could not resolve hostname classes: Name or service not known
slave1: starting namenode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-namenode-slave1.out
master: starting namenode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-namenode-master.out
09:45:30,412: ssh: Could not resolve hostname 09:45:30,412: Name or service not known
to: ssh: Could not resolve hostname to: No address associated with hostname
slave2: starting datanode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-datanode-slave2.out
slave1: starting datanode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-datanode-slave1.out
Starting secondary namenodes [2018-10-15 09:45:43,878 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable]
util.NativeCodeLoader: ssh: Could not resolve hostname util.nativecodeloader: Name or service not known
WARN: ssh: Could not resolve hostname warn: Name or service not known
[main]: ssh: Could not resolve hostname [main]: Name or service not known
2018-10-15: ssh: Could not resolve hostname 2018-10-15: Name or service not known
-: ssh: Could not resolve hostname -: Name or service not known
(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (nativecodeloader.java:<clinit>(62)): Name or service not known
Unable: ssh: Could not resolve hostname unable: Name or service not known
09:45:43,878: ssh: Could not resolve hostname 09:45:43,878: Name or service not known
to: ssh: Could not resolve hostname to: No address associated with hostname
native-hadoop: ssh: Could not resolve hostname native-hadoop: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
for: ssh: Could not resolve hostname for: Name or service not known
platform...: ssh: Could not resolve hostname platform...: Name or service not known
your: ssh: Could not resolve hostname your: Name or service not known
builtin-java: ssh: Could not resolve hostname builtin-java: Name or service not known
classes: ssh: Could not resolve hostname classes: Name or service not known
where: ssh: Could not resolve hostname where: Name or service not known
applicable: ssh: Could not resolve hostname applicable: Name or service not known
using: ssh: Could not resolve hostname using: Name or service not known
load: ssh: Could not resolve hostname load: Name or service not known
通过报错现象分析,是由于ssh解析错误,但是我的/etc/hosts里面主机名与ip地址的映射关系完全没问题,这个时候就不能是ssh问题,那就肯定是hadoop本身的问题。
查看上面的图,看我红框里面的信息,不难发现ssh把
Starting namenodes on [2018-10-15 09:45:30,412 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
master slave1]
这条警告信息当做域名去解析了,所以肯定有问题。
解决办法:
打开 hadoop路径/etc/hadoop/log4j.properties 添加如下信息:
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
这条信息的作用就是在log4j里面去除警告,记住是添加。
网上解决其他办法:(对于我这种情况不合适)
出现上述问题主要是环境变量没设置好,在~/.bash_profile或者/etc/profile中加入以下语句就没问题了。
#vi /etc/profile或者vi ~/.bash_profile
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib",
有的答案说不能加这个双引号:export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib,但具体怎样,我也不太清楚,因为这种方法没有解决掉我问题。所以说很多问题虽然看似跟网上其他人的问题相似,但最终却不能解决掉自己问题,在这种情况下还得自己好好分析,分析出错的现象,说实话,我这个问题我也是花了不少时间才发现的,也是由于不细心以及太依赖于百度了。这里记录一下,也是希望可以帮助到出现类似问题的小伙伴。
然后用source重新编译使之生效即可!
#source /etc/profile或者source ~/.bash_profile
还有一种情况也会出现类似的警告问题,那就是版本不兼容。比如说你下载的hadoop版本是64位的,但你的系统是32位,或者相反,系统64位,hadoop32位。
查看hadoop版本:
进入到如下目录:/usr/local/hadoop-2.9.1/lib/native(这是我的hadoop安装目录,具体情况看你自己安装的目录),
使用ldd或者file命令查看版本:centos系统查看
说明是64位的。
Ubuntu查看:
说明是64位。
file命令也可以查看:
同样说明hadoop是64位。查看操作系统版本使用uname -a就可以。
解决办法同样是在log4j的文件里面加入
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
参考:https://blog.csdn.net/l1028386804/article/details/51538611