• Hadoop 2.7.3 完全分布式维护-动态增加datanode篇


    原有环境

    http://www.cnblogs.com/ilifeilong/p/7406944.html

     IP       host JDK linux hadop role
    172.16.101.55 sht-sgmhadoopnn-01 1.8.0_111 CentOS release 6.5 hadoop-2.7.3 NameNode,SecondaryNameNode,ResourceManager
    172.16.101.58 sht-sgmhadoopdn-01 1.8.0_111 CentOS release 6.5 hadoop-2.7.3 DataNode,NodeManager
    172.16.101.59 sht-sgmhadoopdn-02 1.8.0_111 CentOS release 6.5 hadoop-2.7.3 DataNode,NodeManager
    172.16.101.60 sht-sgmhadoopdn-03 1.8.0_111 CentOS release 6.5 hadoop-2.7.3 DataNode,NodeManager
    172.16.101.66 sht-sgmhadoopdn-04 1.8.0_111 CentOS release 6.5 hadoop-2.7.3 DataNode,NodeManager

    现计划向集群新增一台datanode,如表格所示

    1. 配置系统环境

    主机名,ssh互信,环境变量等

    2. 修改namenode节点的slave文件,增加新节点信息

    $ cat slaves 
    sht-sgmhadoopdn-01
    sht-sgmhadoopdn-02
    sht-sgmhadoopdn-03
    sht-sgmhadoopdn-04

    3. 在namenode节点上,将hadoop-2.7.3复制到新节点上,并在新节点上删除data和logs目录中的文件

    $ hostname 
    sht-sgmhadoopnn-01
    $ rsync -az --progress /usr/local/hadoop-2.7.3/* hduser@sht-sgmhadoopdn-04:/usr/local/hadoop-2.7.3/
    
    $ hostname 
    sht-sgmhadoopdn-04
    $ rm -rf /usr/local/hadoop-2.7.3/logs/*
    $ rm -rf /usr/local/hadoop-2.7.3/data/*

    4. 启动新datanode的datanode进程

    $ hadoop-daemon.sh start datanode
    starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hduser-datanode-sht-sgmhadoopdn-04.out
    $ jps
    31875 Jps
    31821 DataNode

     5. 在namenode查看当前集群情况,确认信节点已经正常加入

    5.1 以命令行方式

    $ hdfs dfsadmin -report
    Configured Capacity: 303324561408 (282.49 GB)
    Present Capacity: 83729309696 (77.98 GB)
    DFS Remaining: 83081265152 (77.38 GB)
    DFS Used: 648044544 (618.02 MB)
    DFS Used%: 0.77%
    Under replicated blocks: 0
    Blocks with corrupt replicas: 0
    Missing blocks: 0
    Missing blocks (with replication factor 1): 0
    
    -------------------------------------------------
    Live datanodes (4):
    
    Name: 172.16.101.66:50010 (sht-sgmhadoopdn-04)
    Hostname: sht-sgmhadoopdn-04
    Decommission Status : Normal
    Configured Capacity: 75831140352 (70.62 GB)
    DFS Used: 24576 (24 KB)
    Non DFS Used: 35573932032 (33.13 GB)
    DFS Remaining: 40257183744 (37.49 GB)
    DFS Used%: 0.00%
    DFS Remaining%: 53.09%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Sep 01 22:50:16 CST 2017
    
    
    Name: 172.16.101.60:50010 (sht-sgmhadoopdn-03)
    Hostname: sht-sgmhadoopdn-03
    Decommission Status : Normal
    Configured Capacity: 75831140352 (70.62 GB)
    DFS Used: 216006656 (206 MB)
    Non DFS Used: 61714608128 (57.48 GB)
    DFS Remaining: 13900525568 (12.95 GB)
    DFS Used%: 0.28%
    DFS Remaining%: 18.33%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Sep 01 22:50:15 CST 2017
    
    
    Name: 172.16.101.59:50010 (sht-sgmhadoopdn-02)
    Hostname: sht-sgmhadoopdn-02
    Decommission Status : Normal
    Configured Capacity: 75831140352 (70.62 GB)
    DFS Used: 216006656 (206 MB)
    Non DFS Used: 62057410560 (57.80 GB)
    DFS Remaining: 13557723136 (12.63 GB)
    DFS Used%: 0.28%
    DFS Remaining%: 17.88%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Sep 01 22:50:14 CST 2017
    
    
    Name: 172.16.101.58:50010 (sht-sgmhadoopdn-01)
    Hostname: sht-sgmhadoopdn-01
    Decommission Status : Normal
    Configured Capacity: 75831140352 (70.62 GB)
    DFS Used: 216006656 (206 MB)
    Non DFS Used: 60249300992 (56.11 GB)
    DFS Remaining: 15365832704 (14.31 GB)
    DFS Used%: 0.28%
    DFS Remaining%: 20.26%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Sep 01 22:50:15 CST 2017

    5.2 以web方式

    6. 在namenoe上设置 hdfs 的负载均衡

    $ hdfs dfsadmin -setBalancerBandwidth 67108864
    Balancer bandwidth is set to 67108864
    $ start-balancer.sh -threshold 5
    starting balancer, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hduser-balancer-sht-sgmhadoopnn-01.out

     7. 查看hdfs负载信息(有时候节点数据量较小,看出来数据量变化,可以上传大文件测试)

    8. 启动新节点的nodemanager进程

    $ yarn-daemon.sh start nodemanager
    starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hduser-nodemanager-sht-sgmhadoopdn-04.out
    $ jps
    32562 NodeManager
    32599 Jps
    31821 DataNode

     

  • 相关阅读:
    下拉框Html.DropDownList 和DropDownListFor 的经常用法
    39个让你受益的HTML5教程
    RapeLay(电车之狼R)的结局介绍 (隐藏结局攻略)
    BP神经网络算法学习
    java实现第七届蓝桥杯取球博弈
    java实现第七届蓝桥杯剪邮票
    java实现第七届蓝桥杯剪邮票
    java实现第七届蓝桥杯剪邮票
    java实现第七届蓝桥杯剪邮票
    java实现第七届蓝桥杯抽签
  • 原文地址:https://www.cnblogs.com/ilifeilong/p/7465397.html
Copyright © 2020-2023  润新知