• [Hadoop]安装

    1 从官网下载hadoop稳定版

    http://www.apache.org/dyn/closer.cgi/hadoop/common/

    2 安装JAVA 参考如下blog

    http://www.cnblogs.com/zhangwenjing/p/3580726.html

    安装JDK

          1、将安装包上传的要安装java的位置。

          2、解压缩：tar -zxvf jdk-7u51-linux-i586.gz

          3、删除：rm -Rf jdk-7u51-linux-i586.gz（节省磁盘空间）

    配置

         #vi /etc/profile

         在最后面添加如下内容：

          JAVA_HOME=/usr/local/java/jdk1.7.0_51

          CLASSPATH=.:＄JAVA_HOME/lib/tools.jar

          PATH=＄JAVA_HOME/bin:＄PATH

          export JAVA_HOME CLASSPATH PATH

          退出保存：按Esc，:wq退出保存。

          验证：java -version

    3 安装hadoop 

    参考http://www.21ops.com/front-tech/9766.html

    其中相关内容具体目录自己改动。

    一：修改配置文件：

    hadoop2.2的配置文件在/opt/hadoop-2.2.0/etc/hadoop文件夹下，具体配置文件修改如下：

    1、修改/etc/hosts文件（sudo gedit /etc/hosts）

    192.168.222.154 hd2-single

    2、修改core-site.xml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    <configuration>
        <property>  
          <name>hadoop.tmp.dir</name>  
          <value>/home/sujx/hadoop/tmp</value>  
        </property>  
        <property>  
          <name>fs.defaultFS</name>  
          <value>hdfs://hd2-single:9000</value>  
          <final>true</final>  
        </property> 
    </configuration>

    fs.defaultFS：HDFS文件系统的URL

    3.  修改hdfs-site.xml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    <configuration>
        <property>  
          <name>dfs.namenode.name.dir</name>  
          <value>file:/home/sujx/hadoop/dfs/name</value>  
          <final>true</final>  
        </property>  
           
        <property>  
          <name>dfs.datanode.data.dir</name>  
          <value>file:/home/sujx/hadoop/dfs/data</value>  
          <final>true</final>  
        </property>  
           
        <property>  
          <name>dfs.replication</name>  
          <value>1</value>  
        </property>  
           
        <property>  
          <name>dfs.permissions</name>  
          <value>false</value>  
        </property>  
    </configuration>

    4.  修改mapred-site.xml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    <configuration>
        <property>  
          <name>mapreduce.framework.name</name>  
          <value>yarn</value>  
        </property>  
           
        <property>  
          <name>mapred.system.dir</name>  
          <value>file:/home/sujx/hadoop/mapred/system</value>  
          <final>true</final>  
        </property>  
           
        <property>  
          <name>mapred.local.dir</name>  
          <value>file:/home/sujx/hadoop/mapred/local</value>  
          <final>true</final>  
        </property> 
    </configuration>

    5.  修改yarn-site.xml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    <configuration>
     
        <property>  
          <name>yarn.nodemanager.aux-services</name>  
          <value>mapreduce_shuffle</value>  
          <description>shuffle service that needsto be set for Map Reduce to run </description>  
        </property>  
           
        <property>  
          <name>yarn.resourcemanager.hostname</name>  
          <value>hd2-single</value>  
          <description>hostanem of RM</description>  
        </property> 
    </configuration>

    6.  修改slave

    hd2-single

    至此，配置文件修改完毕，比较多，挺麻烦的。

    二：启动Hadoop脚本。

    启动hadoop脚本，需呀用到一些环境变量，所以需要先修改Ubuntu的profile文件。

    使用命令：sudo /etc/profile

    1
    2
    3
    4
    5
    6
    7
    8
    export HADOOP_HOME=/opt/hadoop-2.2.0  
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

    在初次运行Hadoop的时候需要初始化Hadoop文件系统，命令如下：

    1
    hdfs namenode -format

    1.  启动脚本一：

    sujx@ubuntu:~$ hadoop-daemon.sh start namenode
    starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.out
    sujx@ubuntu:~$ hadoop-daemon.sh start datanode
    starting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.out
    sujx@ubuntu:~$ hadoop-daemon.sh start secondarynamenode
    starting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.out
    sujx@ubuntu:~$ jps
    9310 SecondaryNameNode
    9345 Jps
    9140 NameNode
    9221 DataNode
    sujx@ubuntu:~$ yarn-daemon.sh start resourcemanager
    starting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.out
    sujx@ubuntu:~$ yarn-daemon.sh start nodemanager
    starting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.out
    sujx@ubuntu:~$ jps
    9310 SecondaryNameNode
    9651 NodeManager
    9413 ResourceManager
    9140 NameNode
    9709 Jps
    9221 DataNode
    sujx@ubuntu:~$

    2.  启动脚本二：

    sujx@ubuntu:~$ start-dfs.sh 
    Starting namenodes on [hd2-single]
    hd2-single: starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.out
    hd2-single: starting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.out
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.out
    sujx@ubuntu:~$ start-yarn.sh 
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.out
    hd2-single: starting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.out
    sujx@ubuntu:~$ jps
    11414 SecondaryNameNode
    10923 NameNode
    11141 DataNode
    12038 Jps
    11586 ResourceManager
    11811 NodeManager
    sujx@ubuntu:~$

    3.  启动脚本三：


    sujx@ubuntu:~$ start-all.sh 
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    Starting namenodes on [hd2-single]
    hd2-single: starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.out
    hd2-single: starting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.out
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.out
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.out
    hd2-single: starting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.out
    sujx@ubuntu:~$ jps
    14156 NodeManager
    14445 Jps
    13267 NameNode
    13759 SecondaryNameNode
    13485 DataNode
    13927 ResourceManager
    sujx@ubuntu:~$

    其实这三种方式最终效果都是相同，他们内部也都是相互调用关系。对应的结束脚本也简单：
     1. 结束脚本一：
    sujx@ubuntu:~$ hadoop-daemon.sh stop nodemanager
    sujx@ubuntu:~$ hadoop-daemon.sh stop resourcemanager
    sujx@ubuntu:~$ hadoop-daemon.sh stop secondarynamenode
    sujx@ubuntu:~$ hadoop-daemon.sh stop datanode
    sujx@ubuntu:~$ hadoop-daemon.sh stop namenode
     2. 结束脚本二：
    sujx@ubuntu:~$ stop-yarn.sh 
    sujx@ubuntu:~$ stop-dfs.sh 
      3. 结束脚本三：
    sujx@ubuntu:~$ stop-all.sh 
     至此，单机伪分布就已经部署完毕。
  • 相关阅读:
    unix改变shell显示颜色
    实习三个月的地一个完整项目总结
    Can&#39;t open named pipe to host: . pipe: MySQL
    OpenGL 与 GLSL 版本号
    牛腩新闻系统(二)——原型图、数据库文档
    【Android开发学习之路】
    JAVA多线程的问题以及处理【转】
    Class.forName的作用以及为什么要用它【转】
    java的集合类【转】
    JAVA的包装类 【转】
  • 原文地址：https://www.cnblogs.com/colipso/p/3924707.html
Copyright © 2020-2023  润新知