• hadooop 配置多网卡 提供跨网段服务


    http://hortonworks.com/blog/multihoming-on-hadoop-yarn-clusters/

    https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html

    公司业务有一个需要就是想把namenode,resourcemanager配置成0.0.0.0:8030之类的方式,这样可以接受

    不同网段的datanode,跨网段服务。

    我找到了以上两个链接的资源,发现是2.6版本才开始支持一些参数,

    我在本地cdh5.0.2里面配置的hadoop2.3是不支持这个版本的,不能认识这些bind-host的参数,

    而公司那个集群用的还是mr1,更是不能接受这个参数。

    不过至少知道了,现在新的版本里这个问题是解决了的。旧的版本里修改后会报错误,请指定dfs.ha.namenode.id ,dfs.nameservice.id之类的。

    The solution is to have separate setting for server endpoints to force binding the wildcard IP address INADDR_ANY i.e. 0.0.0.0. Do NOT supply a port number with any of these settings.
    
    <property>
      <name>dfs.namenode.rpc-bind-host</name>
      <value>0.0.0.0</value>
      <description>
        The actual address the RPC server will bind to. If this optional address is
        set, it overrides only the hostname portion of dfs.namenode.rpc-address.
        It can also be specified per name node or name service for HA/Federation.
        This is useful for making the name node listen on all interfaces by
        setting it to 0.0.0.0.
      </description>
    </property>
    
    <property>
      <name>dfs.namenode.servicerpc-bind-host</name>
      <value>0.0.0.0</value>
      <description>
        The actual address the service RPC server will bind to. If this optional address is
        set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
        It can also be specified per name node or name service for HA/Federation.
        This is useful for making the name node listen on all interfaces by
        setting it to 0.0.0.0.
      </description>
    </property>
    
    <property>
      <name>dfs.namenode.http-bind-host</name>
      <value>0.0.0.0</value>
      <description>
        The actual adress the HTTP server will bind to. If this optional address
        is set, it overrides only the hostname portion of dfs.namenode.http-address.
        It can also be specified per name node or name service for HA/Federation.
        This is useful for making the name node HTTP server listen on all
        interfaces by setting it to 0.0.0.0.
      </description>
    </property>
    
    <property>
      <name>dfs.namenode.https-bind-host</name>
      <value>0.0.0.0</value>
      <description>
        The actual adress the HTTPS server will bind to. If this optional address
        is set, it overrides only the hostname portion of dfs.namenode.https-address.
        It can also be specified per name node or name service for HA/Federation.
        This is useful for making the name node HTTPS server listen on all
        interfaces by setting it to 0.0.0.0.
      </description>
    </property>
    
    Clients use Hostnames when connecting to DataNodes
    
    By default HDFS clients connect to DataNodes using the IP address provided by the NameNode. Depending on the network configuration this IP address may be unreachable by the clients. The fix is letting clients perform their own DNS resolution of the DataNode hostname. The following setting enables this behavior.
    
    <property>
      <name>dfs.client.use.datanode.hostname</name>
      <value>true</value>
      <description>Whether clients should use datanode hostnames when
        connecting to datanodes.
      </description>
    </property>
    
    DataNodes use HostNames when connecting to other DataNodes
    
    Rarely, the NameNode-resolved IP address for a DataNode may be unreachable from other DataNodes. The fix is to force DataNodes to perform their own DNS resolution for inter-DataNode connections. The following setting enables this behavior.
    
    <property>
      <name>dfs.datanode.use.datanode.hostname</name>
      <value>true</value>
      <description>Whether datanodes should use datanode hostnames when
        connecting to other datanodes for data transfer.
      </description>
    </property>
    
  • 相关阅读:
    cdn jsdelivr + github releases 以wordpress sakura主题manifest为例
    c++后台开发面试常见知识点总结(四)数据库
    c++后台开发面试常见知识点总结(三)操作系统
    c++后台开发面试常见知识点总结(二)网络编程
    c++后台开发面试常见知识点总结(一)c++基础
    [LeetCode 206] Reverse Linked List 翻转单链表
    java.util.concurrent.BlockingQueue指南
    如何定义Spring Boot Filter
    Spring Web MVC Framework mvc框架
    springboot profile 日志配置
  • 原文地址:https://www.cnblogs.com/huaxiaoyao/p/4996516.html
Copyright © 2020-2023  润新知