• hadoop 2.x HA 出现ssh不能解析问题记录。


    在docker里面安装hadoop HA 在启动或者停止的时候报ssh不能解析问题。

    问题现象:

    发现图片不清晰:把问题现象粘贴如下:

    root@master:/usr/local/hadoop-2.9.1# start-dfs.sh
    Starting namenodes on [2018-10-15 09:45:30,412 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    master slave1]
    [main]: ssh: Could not resolve hostname [main]: Name or service not known
    (NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (nativecodeloader.java:<clinit>(62)): Name or service not known
    WARN: ssh: Could not resolve hostname warn: Name or service not known
    -: ssh: Could not resolve hostname -: Name or service not known
    Unable: ssh: Could not resolve hostname unable: Name or service not known
    load: ssh: Could not resolve hostname load: Name or service not known
    native-hadoop: ssh: Could not resolve hostname native-hadoop: Name or service not known
    for: ssh: Could not resolve hostname for: Name or service not known
    2018-10-15: ssh: Could not resolve hostname 2018-10-15: Name or service not known
    library: ssh: Could not resolve hostname library: Name or service not known
    platform...: ssh: Could not resolve hostname platform...: Name or service not known
    your: ssh: Could not resolve hostname your: Name or service not known
    using: ssh: Could not resolve hostname using: Name or service not known
    where: ssh: Could not resolve hostname where: Name or service not known
    builtin-java: ssh: Could not resolve hostname builtin-java: Name or service not known
    util.NativeCodeLoader: ssh: Could not resolve hostname util.nativecodeloader: Name or service not known
    applicable: ssh: Could not resolve hostname applicable: Name or service not known
    classes: ssh: Could not resolve hostname classes: Name or service not known
    slave1: starting namenode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-namenode-slave1.out
    master: starting namenode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-namenode-master.out
    09:45:30,412: ssh: Could not resolve hostname 09:45:30,412: Name or service not known
    to: ssh: Could not resolve hostname to: No address associated with hostname
    slave2: starting datanode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-datanode-slave2.out
    slave1: starting datanode, logging to /usr/local/hadoop-2.9.1/logs/hadoop-root-datanode-slave1.out
    Starting secondary namenodes [2018-10-15 09:45:43,878 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable]
    util.NativeCodeLoader: ssh: Could not resolve hostname util.nativecodeloader: Name or service not known
    WARN: ssh: Could not resolve hostname warn: Name or service not known
    [main]: ssh: Could not resolve hostname [main]: Name or service not known
    2018-10-15: ssh: Could not resolve hostname 2018-10-15: Name or service not known
    -: ssh: Could not resolve hostname -: Name or service not known
    (NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (nativecodeloader.java:<clinit>(62)): Name or service not known
    Unable: ssh: Could not resolve hostname unable: Name or service not known
    09:45:43,878: ssh: Could not resolve hostname 09:45:43,878: Name or service not known
    to: ssh: Could not resolve hostname to: No address associated with hostname
    native-hadoop: ssh: Could not resolve hostname native-hadoop: Name or service not known
    library: ssh: Could not resolve hostname library: Name or service not known
    for: ssh: Could not resolve hostname for: Name or service not known
    platform...: ssh: Could not resolve hostname platform...: Name or service not known
    your: ssh: Could not resolve hostname your: Name or service not known
    builtin-java: ssh: Could not resolve hostname builtin-java: Name or service not known
    classes: ssh: Could not resolve hostname classes: Name or service not known
    where: ssh: Could not resolve hostname where: Name or service not known
    applicable: ssh: Could not resolve hostname applicable: Name or service not known
    using: ssh: Could not resolve hostname using: Name or service not known
    load: ssh: Could not resolve hostname load: Name or service not known

    通过报错现象分析,是由于ssh解析错误,但是我的/etc/hosts里面主机名与ip地址的映射关系完全没问题,这个时候就不能是ssh问题,那就肯定是hadoop本身的问题。

    查看上面的图,看我红框里面的信息,不难发现ssh把

    Starting namenodes on [2018-10-15 09:45:30,412 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    master slave1]

    这条警告信息当做域名去解析了,所以肯定有问题。

    解决办法:

    打开 hadoop路径/etc/hadoop/log4j.properties 添加如下信息:

    log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
    这条信息的作用就是在log4j里面去除警告,记住是添加。

    网上解决其他办法:(对于我这种情况不合适)

    出现上述问题主要是环境变量没设置好,在~/.bash_profile或者/etc/profile中加入以下语句就没问题了。

      #vi /etc/profile或者vi ~/.bash_profile
        export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
        export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib",

    有的答案说不能加这个双引号:export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib,但具体怎样,我也不太清楚,因为这种方法没有解决掉我问题。所以说很多问题虽然看似跟网上其他人的问题相似,但最终却不能解决掉自己问题,在这种情况下还得自己好好分析,分析出错的现象,说实话,我这个问题我也是花了不少时间才发现的,也是由于不细心以及太依赖于百度了。这里记录一下,也是希望可以帮助到出现类似问题的小伙伴。

    然后用source重新编译使之生效即可!
      #source /etc/profile或者source ~/.bash_profile

    还有一种情况也会出现类似的警告问题,那就是版本不兼容。比如说你下载的hadoop版本是64位的,但你的系统是32位,或者相反,系统64位,hadoop32位。

    查看hadoop版本:

    进入到如下目录:/usr/local/hadoop-2.9.1/lib/native(这是我的hadoop安装目录,具体情况看你自己安装的目录),

    使用ldd或者file命令查看版本:centos系统查看

    说明是64位的。

    Ubuntu查看:

    说明是64位。

    file命令也可以查看:

    同样说明hadoop是64位。查看操作系统版本使用uname -a就可以。

    解决办法同样是在log4j的文件里面加入

    log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR



    参考:https://blog.csdn.net/l1028386804/article/details/51538611
    记录学习和生活的酸甜苦辣.....哈哈哈
  • 相关阅读:
    103
    101
    102
    100
    ByteView和Sink
    二叉排序树删除、搜索、插入的迭代实现
    怎样就地反转单链表?
    有序单链表的合并
    有序数组的合并
    静态表之整型数组的插入、删除、查找
  • 原文地址:https://www.cnblogs.com/yjt1993/p/9789761.html
Copyright © 2020-2023  润新知