• CentOS系统下Hadoop 2.4.1集群安装配置(简易版)


    安装配置

    1、软件下载

    JDK下载:jdk-7u65-linux-i586.tar.gz

    http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

    Hadoop下载:hadoop-2.4.1.tar.gz

    http://www.apache.org/dyn/closer.cgi/hadoop/common/

    2、/etc/hosts配置

    1. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4  
    2. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6  
    3.   
    4. <strong><span style="color:#ff0000;">192.168.1.2 Master.Hadoop  
    5. 192.168.1.3 Slave1.Hadoop</span></strong>  

    3、/etc/profile配置

    1. export JAVA_HOME=/usr/java/jrockit-jdk1.6.0_45-R28.2.7-4.1.0  
    2. export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib  
    3. export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin  
    4.   
    5. export HADOOP_HOME=/usr/hadoop  
    6. export HADOOP_HOME_WARN_SUPPRESS=1  
    7. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin  

    4、~/etc/hadoop/core-site.xml配置

    1. <configuration>  
    2.       
    3.         <property>   
    4.            <name>fs.defaultFS</name>   
    5.            <value>hdfs://Master.Hadoop:9000</value>   
    6.            <description>   
    7.               Where to find the Hadoop Filesystem through the network.   
    8.               Note 9000 is not the default port.   
    9.               (This is slightly changed from previous versions which didnt have "hdfs")   
    10.            </description>   
    11.    </property>   
    12.      
    13.    <property>  
    14.             <name>hadoop.tmp.dir</name>  
    15.             <value>/usr/hadoop/tmp</value>  
    16.     </property>  
    17.       
    18. </configuration>  

    5、~/etc/hadoop/mapred-site.xml配置

    1. <configuration>      
    2.     <property>   
    3.         <name>mapreduce.framework.name</name>   
    4.     <value>yarn</value>   
    5.     </property>  
    6. </configuration>  

    6、etc/hadoop/yarn-site.xml配置

    1. <configuration>  
    2.       
    3.     <property>      
    4.     <name>yarn.resourcemanager.scheduler.address</name>    
    5.     <value>Master.Hadoop:8030</value>   
    6.     </property>  
    7.   
    8.     <property>      
    9.         <name>yarn.resourcemanager.resource-tracker.address</name>     
    10.     <value>Master.Hadoop:8031</value>   
    11.     </property>  
    12.       
    13.     <property>    
    14.     <name>yarn.resourcemanager.address</name>       
    15.     <value>Master.Hadoop:8032</value>    
    16.     </property>  
    17.       
    18.     <property>    
    19.     <name>yarn.resourcemanager.admin.address</name>    
    20.     <value>Master.Hadoop:8033</value>     
    21.     </property>   
    22.         
    23.    <property>      
    24.     <name>yarn.resourcemanager.webapp.address</name>      
    25.     <value>Master.Hadoop:8088</value>    
    26.     </property>  
    27.       
    28.     <property>      
    29.     <name>yarn.resourcemanager.webapp.https.address</name>      
    30.     <value>Master.Hadoop:8090</value>    
    31.     </property>  
    32.       
    33.     <property>    
    34.     <name>yarn.nodemanager.local-dirs</name>    
    35.     <value>${hadoop.tmp.dir}/nodemanager/local</value>    
    36.     <description>the local directories used by the nodemanager</description>    
    37.   </property>  
    38.     
    39.   <property>    
    40.     <name>yarn.nodemanager.remote-app-log-dir</name>    
    41.     <value>${hadoop.tmp.dir}/nodemanager/remote</value>    
    42.     <description>directory on hdfs where the application logs are moved to </description>    
    43.   </property>  
    44.     
    45.   <property>    
    46.     <name>yarn.nodemanager.log-dirs</name>    
    47.     <value>${hadoop.tmp.dir}/nodemanager/logs</value>    
    48.     <description>the directories used by Nodemanagers as log directories</description>    
    49.   </property>  
    50.     
    51.   <property>    
    52.     <name>yarn.nodemanager.aux-services</name>    
    53.     <value>mapreduce_shuffle</value>    
    54.     <description>shuffle service that needs to be set for Map Reduce to run </description>    
    55.   </property>  
    56.     
    57.   <property>      
    58.     <name>mapreduce.jobhistory.address</name>     
    59.     <value>Master.Hadoop:10020</value>   
    60.     </property>  
    61.          
    62.     <property>      
    63.     <name>mapreduce.jobhistory.webapp.address</name>     
    64.     <value>Master.Hadoop:19888</value>   
    65.     </property>  
    66.       
    67. </configuration>  

    7、~/etc/hadoop/hdfs-site.xml配置

    1. <configuration>  
    2.       
    3.     <property>    
    4.    <name>dfs.permissions.superusergroup</name>    
    5.    <value>root</value>   
    6.     </property>  
    7.       
    8.     <property>   
    9.     <name>dfs.replication</name>   
    10.     <value>1</value>   
    11.   </property>   
    12.     
    13. </configuration>  

    启动与验证

    1、格式化HDFS文件系统

    hadoop namenode -format

    2、启动Hadoop

    启动前关闭集群中所有机器的防火墙

    service iptables stop

    启动命令

    start-all.sh

    3、验证Hadoop

    方式一:jps

    方式二:hadoop dfsadmin -report

    ZOOM 云视频会议网站:http://www.zoomonline.cn/

  • 相关阅读:
    1001.A+B Format(20)
    大一下学期的自我目标
    re模块3
    re模块2
    re模块
    configParser模块
    logging模块
    hashlib模块
    sys模块
    isinstance函数
  • 原文地址:https://www.cnblogs.com/gw811/p/4608088.html
Copyright © 2020-2023  润新知