• 在Ubuntu14.10中部署Hadoop2.6.0单节点伪分布集群


    1. 环境信息如下:

    ubuntu:14.10

    jdk:openjdk-1.7.0

    hadoop:2.6.0

    2. 下载hadoop2.6.0, http://apache.fayea.com/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

    本文的$HADOOP_HOME为:/media/GoldenResources/localSoft/hadoop-2.6.0。

    在 ~/.bashrc文件中添加环境变量,如下:

    export HADOOP_HOME=/media/GoldenResources/localSoft/hadoop-2.6.0
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    

    然后使配置生效,命令如下: 

    source ~/.bashrc
    

    3. 安装JDK,并设置JAVA_HOME环境变量。在/etc/profile文件最后添加如下内容

    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386   //根据自己Java安装目录而定
    export PATH=$JAVA_HOME/bin:$PATH
    

    输入以下命令使配置生效

    source /etc/profile
    

    4. 配置SSH。首先生成秘钥,命令如下,然后一路回车确认,不需要任何输入。 

    ssh-keygen -t rsa
    

    然后把公钥写入authorized_keys文件中,命令如下:

    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    

    最后,输入下述命令,按提示输入 yes 即可,验证无密码访问是否成功。

    ssh localhost 
    

    5. 修改Hadoop配置文件,进入${HADOOP_HOME}/etc/hadoop/目录。
    1)修改core-site.xml,添加如下内容。

    <property>
    	<name>fs.default.name</name>
    	<value>hdfs://localhost:9000</value>		
    </property>
    

    2)修改hdfs-site.xml,添加以下内容。

    <property>
        <name>dfs.repliacation</name>
         <value>1</value>
    </property>
    

    3)将mapred-site.xml.template重命名为mapred-site.xml,并添加如下内容。

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    

    4)修改yarn-site.xml,添加以下内容。

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    

    6. 格式化HDFS,命令如下:

    bin/hdfs namenode -format 注释:bin/hadoop namenode -format命令已过时
    

    默认情况下,格式化成功会在/tmp/hadoop-${user.name}目录下创建dfs文件夹。

    7.启动HDFS,命令如下:

    sbin/start-dfs.sh 
    

    出现以下错误:

    $sbin/start-dfs.sh 
    15/03/16 21:36:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Starting namenodes on [OpenJDK Server VM warning: You have loaded library /media/GoldenResources/localSoft/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
    localhost]
    sed: -e 表达式 #1, 字符 8: “s”的未知选项↵
    Server: ssh: Could not resolve hostname server: Name or service not known
    -c: Unknown cipher type 'cd'
    disabled: ssh: Could not resolve hostname disabled: Name or service not known
    or: ssh: Could not resolve hostname or: Name or service not known
    which: ssh: Could not resolve hostname which: Name or service not known
    now.: ssh: Could not resolve hostname now.: Name or service not known
    will: ssh: Could not resolve hostname will: Name or service not known
    fix: ssh: Could not resolve hostname fix: Name or service not known
    with: ssh: Could not resolve hostname with: Name or service not known
    with: ssh: Could not resolve hostname with: Name or service not known
    'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
    recommended: ssh: Could not resolve hostname recommended: Name or service not known
    that: ssh: Could not resolve hostname that: Name or service not known
    OpenJDK: ssh: Could not resolve hostname openjdk: Name or service not known
    noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
    '-z: ssh: Could not resolve hostname '-z: Name or service not known
    It's: ssh: Could not resolve hostname it's: Name or service not known
    <libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
    it: ssh: Could not resolve hostname it: Name or service not known
    link: ssh: Could not resolve hostname link: Name or service not known
    have: ssh: Could not resolve hostname have: Name or service not known
    have: ssh: Could not resolve hostname have: Name or service not known
    

    参考 http://blog.csdn.net/xin_jmail/article/details/40556267中的解决方法,设置 HADOOP_COMMON_LIB_NATIVE_DIR和HADOOP_OPTS环境变量,在 ~/.bashrc文件中添加如下内容并编译。

    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"  
    

     source ~/.bashrc

    重新启动HDFS,输出如下,表示启动成功。

    $start-dfs.sh 
    15/03/16 22:21:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Starting namenodes on [localhost]
    localhost: starting namenode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-namenode-qunengrong-Studio-1450.out
    localhost: starting datanode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-datanode-qunengrong-Studio-1450.out
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: starting secondarynamenode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-secondarynamenode-qunengrong-Studio-1450.out
    15/03/16 22:21:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    

    可通过http://localhost:50070/ 来查看NameNode的运行状态

    8. 启动Yarn,命令如下:

    sbin/start-yarn.sh
    

    可通过http://localhost:8088/cluster 来查看Application的执行状态

  • 相关阅读:
    高性能MySQL--创建高性能的索引
    error:0906D064:PEM routines:PEM_read_bio:bad base64 decode
    高性能MySQL--MySQL数据类型介绍和最优数据类型选择
    Elasticsearch入门和查询语法分析(ik中文分词)
    裁员浪潮,互联网人该何去何从?
    django+mysql的使用
    很详细的Django入门详解
    (2021年1月5日亲测有效)最新PyCharm 安装教程&激活破解,Pycharm激活,Pycharm破解
    (2021年1月5日更新)!最新的pycharm永久激活办法,亲测有效
    太干了!一张图整理了 Python 所有内置异常
  • 原文地址:https://www.cnblogs.com/Cherise/p/4343220.html
Copyright © 2020-2023  润新知