• Hadoop 2.7.4 HDFS+YRAN HA删除datanode和nodemanager

    当前集群

    主机名称 IP地址 角色 统一安装目录 统一安装用户
    sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager

    /usr/local/hadoop(软连接)

    /usr/local/hadoop-2.7.4

    /usr/local/zookeeper（软连接）

    /usr/local/zookeeper-3.4.9

    root
    sht-sgmhadoopnn-02 172.16.101.56 namenode,resourcemanager
    sht-sgmhadoopdn-01 172.16.101.58 datanode,nodemanager,journalnode,zookeeper
    sht-sgmhadoopdn-02 172.16.101.59 datanode,nodemanager,journalnode,zookeeper
    sht-sgmhadoopdn-03 172.16.101.60 datanode,nodemanager,journalnode,zookeeper
    sht-sgmhadoopdn-04 172.16.101.66 datanode,nodemanager

    因sht-sgmhadoopdn-04节点异常，需要从当前集群剔除

    1. 在active节点sht-sgmhadoopnn-01修改配置文件

    slaves

    sht-sgmhadoopdn-01
sht-sgmhadoopdn-02
sht-sgmhadoopdn-03

    hdfs-site.xml

      <property>
        <name>dfs.replication</name>
        <value>3</value>
  </property>

    hdfs-site.xml

      <property>
        <name>dfs.hosts</name>
        <value>/usr/local/hadoop/etc/hadoop/hdfs_includes</value>
  </property>

  <property>
        <name>dfs.hosts.exclude</name>
        <value>/usr/local/hadoop/etc/hadoop/hdfs_excludes</value>
  </property>

    yarn-site.xml

      <property>
        <name>dfs.hosts</name>
        <value>/usr/local/hadoop/etc/hadoop/yarn_includes</value>
  </property>

  <property>
        <name>dfs.hosts.exclude</name>
        <value>/usr/local/hadoop/etc/hadoop/yarn_excludes</value>
  </property>

    如果存在上述hosts配置，则可以直接进入第二步，否则，需要将上述参数添加至对应文件里，并重启hdfs和yarn集群

     2. 添加白名单和黑名单

    白名单

    # echo sht-sgmhadoopdn-01 > /usr/local/hadoop/etc/hadoop/hdfs_includes
# echo sht-sgmhadoopdn-02 > /usr/local/hadoop/etc/hadoop/hdfs_includes
# echo sht-sgmhadoopdn-03 > /usr/local/hadoop/etc/hadoop/hdfs_includes

    黑名单

    # echo sht-sgmhadoopdn-04 > /usr/local/hadoop/etc/hadoop/hdfs_excludes
# echo sht-sgmhadoopdn-04 > /usr/local/hadoop/etc/hadoop/yarn_excludes

    在所有节点更新上述参数

    3. 在active节点sht-sgmhadoopnn-01刷新节点信息

    此命令告诉resourcemanager重新读取配置文件yarn-site.xml，将存在yarn_excludes文件中的节点退出当前yarn集群，并不在分配job到该节点

    # yarn rmadmin -refreshNodes

    输出log

    2019-03-30 00:40:48,751 INFO org.apache.hadoop.conf.Configuration: found resource yarn-site.xml at file:/usr/local/hadoop-2.7.4/etc/hadoop/yarn-site.xml
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Setting the includes file to /usr/local/hadoop/etc/hadoop/yarn_includes
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Setting the excludes file to /usr/local/hadoop/etc/hadoop/yarn_excludes
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-01 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-02 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes
2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-03 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes
2019-03-30 00:40:48,756 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-04 to the list of excluded hosts from /usr/local/hadoop/etc/hadoop/yarn_excludes
2019-03-30 00:40:48,757 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node sht-sgmhadoopdn-04:21570 as it is now DECOMMISSIONED
2019-03-30 00:40:48,758 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: sht-sgmhadoopdn-04:21570 Node Transitioned from RUNNING to DECOMMISSIONED
2019-03-30 00:40:48,756 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root	IP=172.16.101.55	OPERATION=refreshNodes	TARGET=AdminService	RESULT=SUCCESS
2019-03-30 00:40:48,772 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Removed node sht-sgmhadoopdn-04:21570 clusterResource: <memory:12288, vCores:24>
2019-03-30 00:40:48,909 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Disallowed NodeManager nodeId: sht-sgmhadoopdn-04:21570 hostname: sht-sgmhadoopdn-04
    View Code

    nodemanager输出log

    2019-03-30 00:40:48,910 WARN org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Recieved SHUTDOWN signal from Resourcemanager as part of heartbeat, hence shutting down.
2019-03-30 00:40:48,910 WARN org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Message from ResourceManager: Disallowed NodeManager nodeId: sht-sgmhadoopdn-04:21570 hostname: sht-sgmhadoopdn-04
2019-03-30 00:40:49,023 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2019-03-30 00:40:49,125 INFO org.apache.hadoop.ipc.Server: Stopping server on 21570
2019-03-30 00:40:49,133 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2019-03-30 00:40:49,133 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 21570
2019-03-30 00:40:49,135 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.
2019-03-30 00:40:49,154 INFO org.apache.hadoop.ipc.Server: Stopping server on 8040
2019-03-30 00:40:49,159 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8040
2019-03-30 00:40:49,159 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2019-03-30 00:40:49,160 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Public cache exiting
2019-03-30 00:40:49,162 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2019-03-30 00:40:49,163 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2019-03-30 00:40:49,163 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2019-03-30 00:40:49,178 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1
2019-03-30 00:40:49,182 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at sht-sgmhadoopdn-04/172.16.101.66
************************************************************/
    View Code

    此命令告诉hdfs重新读取配置文件hdfs-site.xml，将存在hdfs_excludes文件中的节点退出当前hdfs集群。

    # hdfs dfsadmin -refreshNodes

     输出log

    2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-01 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes
2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-02 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes
2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-03 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes
2019-03-30 00:43:00,577 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-04 to the list of excluded hosts from /usr/local/hadoop/etc/hadoop/hdfs_excludes
2019-03-30 00:43:01,266 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.sendHeartbeat from 172.16.101.66:16197 Call#1641 Retry#0
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode because the host is not in the include-list: 172.16.101.66:50010
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1340)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:4512)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1298)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:114)
	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28754)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
2019-03-30 00:43:28,046 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* removeDeadDatanode: lost heartbeat from 172.16.101.66:50010
2019-03-30 00:43:28,051 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default-rack/172.16.101.66:50010
    View Code

    datanode输出log

    2019-03-30 00:43:01,270 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-698223843-172.16.101.55-1553701973789 (Datanode Uuid 30bc4855-5dca-412a-bf9a-9529bcfe3ea0) service to sht-sgmhadoopnn-01/172.16.101.55:8020 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode because the host is not in the include-list: 172.16.101.66:50010
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1340)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:4512)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1298)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:114)
	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28754)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

	at org.apache.hadoop.ipc.Client.call(Client.java:1476)
	at org.apache.hadoop.ipc.Client.call(Client.java:1413)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy15.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:152)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:402)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:500)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659)
	at java.lang.Thread.run(Thread.java:745)
2019-03-30 00:43:01,272 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-698223843-172.16.101.55-1553701973789 (Datanode Uuid 30bc4855-5dca-412a-bf9a-9529bcfe3ea0) service to sht-sgmhadoopnn-01/172.16.101.55:8020
    View Code

    4. 执行hdfs集群负载均衡

    # hdfs dfs -setrep -R -w 3  /

    输出log

    # hdfs dfs -setrep -R -w 3  /
Replication 3 set: /CentOS-6.8-x86_64-bin-DVD2.iso
Replication 3 set: /hadoop-2.8.1.tar.gz
Replication 3 set: /slaves
Replication 3 set: /yarn-env.sh
Waiting for /CentOS-6.8-x86_64-bin-DVD2.iso ...
WARNING: the waiting time may be long for DECREASING the number of replications.
............................................................................................... done
Waiting for /hadoop-2.8.1.tar.gz ... done
Waiting for /slaves ... done
Waiting for /yarn-env.sh ... done
    View Code

    5.停止sht-sgmhadoopdn-04节点的datanode和nodemanager角色

    6. 清空各节点黑白名单

    # echo > /usr/local/hadoop/etc/hadoop/hdfs_excludes
# echo > /usr/local/hadoop/etc/hadoop/yarn_excludes
# echo > /usr/local/hadoop/etc/hadoop/hdfs_includes
# echo > /usr/local/hadoop/etc/hadoop/yarn_includes

     7. 在适当时候建议重启hdfs和yarn集群
  • 相关阅读:
    仿发送微博限制只发送140个字(一个汉字占位2字节)
    Linux下多任务间通信和同步-信号
    Color
    面试遇到的问题
    Jquery Mobile左右滑动效果
    C++ primer（八）--内联函数 引用变量 引用传递函数参数 函数重载/模板/模板具体化
    Python3 配置文件 解析
    Python MySQL ORM QuickORM hacking
    Python 获取 网卡 MAC 地址
    AttributeError: 'dict_values' object has no attribute 'translate'
  • 原文地址：https://www.cnblogs.com/ilifeilong/p/10624198.html
Copyright © 2020-2023  润新知