• Hadoop入门(一):Hadoop伪分布安装


    1安装Hadoop

    首先解压下载来的hadoop 0.20包到/home/admin目录:

    tar xzfhadoop-0.20.2.tar.gz

    配置Hadoop环境变量:

    exportHADOOP_INSTALL=/home/admin/hadoop-0.20.2

    exportPATH=$PATH:$HADOOP_INSTALL/bin

    测试下是否安装成功:

    hadoop version


    2创建SSH无密码模式密钥

    为当前用户配置无密码的SSH登录:

             ssh-keygen-t rsa -P '' -f ~/.ssh/id_rsa

    cat~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

    测试一下是否还提示输入密码:

    ssh localhost


    3配置Hadoop伪分布模式

    /home/admin/hadoop-0.20.2/conf/core-site.xml

    ===============================================================================

    <configuration>

           <property>

                   <name>fs.default.name</name>

                   <value>hdfs://localhost</value>

           </property>

    </configuration>

    /home/admin/hadoop-0.20.2/conf/hdfs-site.xml

    ===============================================================================

    <configuration>

           <property>

                   <name>dfs.replication</name>

                    <value>1</value>

           </property>

    </configuration>

    /home/admin/hadoop-0.20.2/conf/mapred-site.xml

    ===============================================================================

    <configuration>

           <property>

                   <name>mapred.job.tracker</name>

                   <value>localhost:8021</value>

           </property>

    </configuration>


    4启动Hadoop服务

    4.1格式化NameNode

    hadoop namenode -format

    4.2启动服务

    start-dfs.sh

    start-mapred.sh

    4.3常见问题

    在namenode启动脚本%HADOOP_HOME%/bin/start-dfs.sh的时候发现datanode报错:

    Error: JAVA_HOMEis not set

    原因是在%HADOOP_HOME%/conf/hadoop-env.sh内缺少JAVA_HOME的定义,只需要在hadoop-env.sh中增加:

    exportJAVA_HOME=/export/servers/jdk1.6.0_25/


    5测试HDFS

    hadoop fs -mkdir books

    hadoop fs -ls .

    hadoop fs -copyFromLocal NOTICE.txthdfs://localhost/user/root/books/NOTICE.txt

    参考资料

    《Hadoop权威指南》

  • 相关阅读:
    Unreal中的python开发
    2022.5.16
    2022.5.18
    2022.5.4
    2022.5.2
    2022.5.6
    2022.5.14
    2022.4.30
    2022.5.10
    2022.5.8
  • 原文地址:https://www.cnblogs.com/xiaomaohai/p/6157741.html
Copyright © 2020-2023  润新知