• HDFS 搭建记录


    1. 三台服务:

    172.17.0.62(namenode)

    172.17.0.68(datanode)

    172.17.0.76(datanode)

    /etc/hosts包含的内容:

    三台都包含的域名映射:(注意:域名中不要有下划线(_),否则 datanode起不来,呵呵:。。)

    172.17.0.76  tomcattomcatmobileplatformtest.bjdv   #不要有下划线(_)

    172.17.0.62  tomcatmobileplatform.bjdv

    172.17.0.68  mobileplatformweblogic.bjdv

    #各个服务器映射自己的主机名

    127.0.0.1 cd6d8c70882c

    2. namenode与datanode配置互相的不用密码的ssh,以及登陆localhost的不用密码的ssh,这里用的rsa

    每台服务器上执行一下:#localhost的ssh

    1)ssh-keygen -t rsa -f /root/.ssh/id_rsa   (连续两下回车,即不设密码)

    2)cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

    把namenode和datanode上的id_rsa.pub拷到对方的/root下,并在对方上执行一下3,4  #namenode与datanode互相的ssh

    3)cat /root/id_rsa.pub >> /root/.ssh/authorized_keys

    4)chmod 0600 ~/.ssh/authorized_keys

    3.下载并拷贝hadoop(本文使用的是hadoop-2.7.2)到三台服务器。

    4.配置文件:

    etc/hadoop/core-site.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at
    
        http://www.apache.org/licenses/LICENSE-2.0
    
      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->
    
    <!-- Put site-specific property overrides in this file. -->
    
    <configuration>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://172.17.0.62:9000</value>
    </property>
    </configuration>

    etc/hadoop/hadoop-env.sh:

    加入JAVA_HOME,HADOOP_PID_DIR

    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
    export HADOOP_PID_DIR=/nodespace/h/pids

    etc/hadoop/hdfs-site.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at
    
        http://www.apache.org/licenses/LICENSE-2.0
    
      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License. See accompanying LICENSE file.
    -->
    
    <!-- Put site-specific property overrides in this file. -->
    
    <configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>/nodespace/namenode</value>
        </property>
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>/nodespace/datanode</value>
        </property>
    </configuration>

    etc/hadoop/slaves:

    172.17.0.68
    172.17.0.76

    namenode服务器上:

    # mkdir /nodespace/namenode (dfs.namenode.name.dir)

    # bin/hadoop namenode -format

    # sbin/start-dfs.sh

    查看日志

    # tail -f logs/hadoop-root-namenode-2ac9303a23f0.log -n 200

    检查服务

    # jps
    27831 Jps
    27353 NameNode
    27584 SecondaryNameNode
    # ./bin/hdfs dfsadmin -report
    Configured Capacity: 20869324800 (19.44 GB)
    Present Capacity: 14557261824 (13.56 GB)
    DFS Remaining: 14557155328 (13.56 GB)
    DFS Used: 106496 (104 KB)
    DFS Used%: 0.00%
    Under replicated blocks: 0
    Blocks with corrupt replicas: 0
    Missing blocks: 0
    Missing blocks (with replication factor 1): 0
    
    -------------------------------------------------
    Live datanodes (2):
    
    Name: 172.17.0.68:50010 (mobileplatformweblogic.bjdv)
    Hostname: localhost
    Decommission Status : Normal
    Configured Capacity: 10434662400 (9.72 GB)
    DFS Used: 49152 (48 KB)
    Non DFS Used: 4756729856 (4.43 GB)
    DFS Remaining: 5677883392 (5.29 GB)
    DFS Used%: 0.00%
    DFS Remaining%: 54.41%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Mon Aug 29 14:55:37 CST 2016
    
    
    Name: 172.17.0.76:50010 (tomcattomcatmobileplatformtest.bjdv)
    Hostname: localhost
    Decommission Status : Normal
    Configured Capacity: 10434662400 (9.72 GB)
    DFS Used: 57344 (56 KB)
    Non DFS Used: 1555333120 (1.45 GB)
    DFS Remaining: 8879271936 (8.27 GB)
    DFS Used%: 0.00%
    DFS Remaining%: 85.09%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Mon Aug 29 14:55:38 CST 2016

    ( NameNode 的webUI信息 似乎不正确,只能看到一个Datanode, http://172.17.0.62:50070/dfshealth.html, 上面的 bin/hdfs dfsadmin -report 是正确的)

    测试:

    # bin/hadoop fs -put LICENSE.txt /

    datanode服务器上: 

    检查服务

    # jps 
    21622 DataNode
    21774 Jps

    日志查看:

    # tail -f logs/hadoop-root-datanode-cd6d8c70882c.log

    datanode上:

    没有压缩的情况下,往hdfs上传文件后在其中的一个datanode上可以看到(如:blk_1073741828),dfs.replication设置为1,所以在所有datanode上只有一份文件。

    # ls -l datanode/current/BP-333745209-172.17.0.62-1472441549732/current/finalized/subdir0/subdir0/
    total 8
    -rw-r--r-- 1 root root 1366 Aug 29 14:55 blk_1073741828
    -rw-r--r-- 1 root root   19 Aug 29 14:55 blk_1073741828_1004.meta

    以上是自己搭建hdfs时测试与总结,有不对的地方,欢迎指正。

  • 相关阅读:
    部署K2 Blackpearl流程时出错(与基础事务管理器的通信失败或Communication with the underlying transaction manager has failed.)
    用SQL命令将查询结果集导出为文本文件
    SQL函数计算两个日期间的工作日天数
    CommonJS模块加载方法
    async 属性
    如何用VS.NET2005调式.asp和.aspx混合的web系统
    对web.config文件的节点进行加解密
    [转]看似简单的问题 静态方法和实例化方法的区别
    单引号(')和双引号(")
    从一个时间段中查找出星期为“Sunday”的日期
  • 原文地址:https://www.cnblogs.com/niulang85/p/5819006.html
Copyright © 2020-2023  润新知