• Hadoop单节点集群

    Hadoop单节点集群

    官方网站:


    环境:
    CentOS 7.1
    hadoop-2.7.2
    jdk-1.8

    说明:仅个人测试学习记录，大神们可以一笑而过

    一.jdk
    jdk版本，hadoop官方推荐HadoopJavaVersions


    二.安装hadoop
    1.创建用户
    useradd -r -m hadoop
    echo hadoop:hd.123|chpasswd
    说明:服务启停都由hadoop用户来操作
    2.调整资源限制
    cat >/etc/security/limits.d/99-hadoop-nproc.conf <<HERE
    hadoop soft fsize unlimited 
    hadoop soft cpu unlimited 
    hadoop soft as unlimited 
    hadoop soft nofile 64000 
    hadoop soft nproc 64000 
    hadoop hard fsize unlimited 
    hadoop hard cpu unlimited 
    hadoop hard as unlimited 
    hadoop hard nofile 64000 
    hadoop hard nproc 64000 
    HERE
    3.安装
    tar -xvf hadoop-2.7.2.tar.gz -C /opt
    mv /opt/hadoop-2.7.2 /opt/hadoop
    chown -R hadoop: /opt/hadoop


    三.环境变量
    cat >/etc/profile.d/hadoop.sh <<HERE

    export JAVA_HOME=/opt/jdk


    export HADOOP_PREFIX=/opt/hadoop

    export HADOOP_HOME=/opt/hadoop

    export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

    export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC"

    export HADOOP_PORTMAP_OPTS="-Xmx1024m"

    export HADOOP_CLIENT_OPTS="-Xmx1024m"

    export HADOOP_PID_DIR=/opt/hadoop/run

    export HADOOP_SECURE_DN_PID_DIR=/opt/hadoop/run

    export HADOOP_LOG_DIR=/opt/hadoop/logs


    export HADOOP_YARN_HOME=/opt/hadoop

    export YARN_PID_DIR=/opt/hadoop/run


    export HADOOP_MAPRED_PID_DIR=/opt/hadoop/run

    HERE

    source /etc/profile



    四.配置HDFS 
    1.修改配置文件
    etc/hadoop/core-site.xml
    Hadoop单节点集群

    etc/hadoop/hdfs-site.xml

    Hadoop单节点集群

    2.ssh密钥认证

    su - hadoop

    ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

    cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

    chmod 0600 ~/.ssh/authorized_keys

     

    说明:$HADOOP_PREFIX/sbin/start-dfs.sh脚本会通过ssh连接调用，所以要密钥认证

    

    

    3.格式化hdfs

    

    

$HADOOP_PREFIX/bin/hdfs namenode -format


    

    

    

    

    4.namenode & datanode

    

    

    $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

    

    $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

    

    
或者

    

    
当配置了ssh密码认证后可以直接通过脚本启动和停止

    

    
启动

    

    
$HADOOP_PREFIX/sbin/start-dfs.sh
    

    

    

    [hadoop@node-a ~]$
$HADOOP_PREFIX/sbin/start-dfs.sh
    

    Starting
namenodes on [localhost]
    

    localhost:
starting namenode, logging to
/opt/hadoop/logs/hadoop-hadoop-namenode-node-a.example.com.out
    

    localhost:
starting datanode, logging to
/opt/hadoop/logs/hadoop-hadoop-datanode-node-a.example.com.out
    

    Starting
secondary namenodes [0.0.0.0]
    

    0.0.0.0:
starting secondarynamenode, logging to
/opt/hadoop/logs/hadoop-hadoop-secondarynamenode-node-a.example.com.out
    

    注意:主机名要能解析，否则启动报错
    


    

    

    停止
    

    

    $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode

    

    $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode

    

    

    
或者

    

    
当配置了ssh密码认证后可以直接通过脚本启动和停止

    

    
$HADOOP_PREFIX/sbin/stop-dfs.sh
    

    

    

    
[hadoop@node-a
~]$ HADOOP_PREFIX/sbin/stop-dfs.sh
    

    
Stopping
namenodes on [localhost]
    

    
localhost: stopping
namenode
    

    
localhost: stopping
datanode
    

    
Stopping
secondary namenodes [0.0.0.0]
    

    
0.0.0.0:
stopping secondarynamenode
    

    

    

    5.NameNode web接口
    

    localhost:50070
    

    
Hadoop单节点集群
    

    

    

    

    

    

    四.配置YARN
    

    1.修改配置文件
    

    etc/hadoop/mapred-site.xml
    

    Hadoop单节点集群
    

    

    

    etc/hadoop/yarn-site.xml
    

Hadoop单节点集群
    


    

    2.ssh密钥认证
    

    

    

    
同上，略

    

    

    

    


    
3.resourcemanager & nodemanager & historyserver

    

    

    

    

    
启动

    

    
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

    

    

    

    $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager

    

    $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver

    


    

    

    $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

    

    

    

    

    
或者

    

    
当配置了ssh密码认证后可以直接通过脚本启动和停止

    

    $HADOOP_PREFIX/sbin/start-yarn.sh
    

    

    

    提示:start-yarn.sh最后一行的start proxyserver默认是注释状态，可以按需启用

    

    $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

    


    

    

    

    
[hadoop@node-a
~]$ $HADOOP_PREFIX/sbin/start-yarn.sh
    

    
starting
yarn daemons
    

    
starting
resourcemanager, logging to
/opt/hadoop/logs/yarn-root-resourcemanager-ct7.example.com.out
    

    
localhost: starting
nodemanager, logging to
/opt/hadoop/logs/yarn-root-nodemanager-ct7.example.com.out
    

    
 
    

    

    

    
停止
    

    

    

    

    
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager

    

    

    

    $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager

    

    $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop proxyserver

    

    

    $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

    

    

    

    
或者

    

    
当配置了ssh密码认证后可以直接通过脚本启动和停止

    

    
$HADOOP_PREFIX/sbin/stop-yarn.sh

    

    

    
$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

    

    

    

    

    

    
[hadoop@node-a
~]$ HADOOP_PREFIX/sbin/stop-yarn.sh
    

    
stopping
yarn daemons
    

    
stopping
resourcemanager
    

    
localhost: stopping
nodemanager
    

    
no
proxyserver to stop
    

    [hadoop@node-a
~]$ netstat -tunlp|grep
java
    

    tcp  
     0
     0
0.0.0.0:8033    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java
    

    tcp  
     0
     0
0.0.0.0:10020    
      0.0.0.0:*
     
     
  LISTEN    
 17493/java
    

    tcp  
     0
     0
0.0.0.0:50020    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java
    

    tcp  
     0
     0
127.0.0.1:9000    
     0.0.0.0:*
     
     
  LISTEN    
 17642/java
    

    tcp  
     0
     0
0.0.0.0:8040    
     
 0.0.0.0:*    
     
    LISTEN  
   17159/java
    

    tcp  
     0
     0
0.0.0.0:50090    
      0.0.0.0:*
     
     
  LISTEN    
 17938/java
    

    tcp  
     0
     0
0.0.0.0:8042    
     
 0.0.0.0:*    
     
    LISTEN  
   17159/java
    

    tcp  
     0
     0
127.0.0.1:41870    
    0.0.0.0:*  
     
      LISTEN
   
 17772/java
    

    tcp  
     0
     0
0.0.0.0:19888
     
    0.0.0.0:*  
     
      LISTEN
   
 17493/java
    

    tcp  
     0
     0
0.0.0.0:10033    
      0.0.0.0:*
     
     
  LISTEN    
 17493/java
    

    tcp  
     0
     0
0.0.0.0:50070
     
    0.0.0.0:*  
     
      LISTEN
   
 17642/java
    

    tcp  
     0
     0
0.0.0.0:8088  
     
   0.0.0.0:*  
     
      LISTEN
   
 17045/java
    

    tcp  
     0
     0
0.0.0.0:50010    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java
    

    tcp  
     0
     0
0.0.0.0:13562    
      0.0.0.0:*
     
     
  LISTEN    
 17159/java
    

    tcp  
     0
     0
0.0.0.0:50075    
      0.0.0.0:*
     
     
  LISTEN    
 17772/java
    

    tcp  
     0
     0
0.0.0.0:8030    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java
    

    tcp  
     0
     0
0.0.0.0:8031    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java
    

    tcp  
     0
     0
0.0.0.0:36864    
      0.0.0.0:*
     
     
  LISTEN    
 17159/java
    

    tcp  
     0
     0
0.0.0.0:8032    
     
 0.0.0.0:*    
     
    LISTEN  
   17045/java
    

    


    

    

    

    

    

    

    4.ResourceManager
web接口
    

    localhost:8088
    

    Hadoop单节点集群
    

    


    

    

    5.JobHistory Server web接口

    

    
localhost:19888
    
Hadoop单节点集群
    


    

    

    

    

    

    							
		

    
    
          
    • 

          
  • 

            相关阅读:
    
            
              Java Concurrency 
                
    Java Annotation,Java注解 
                
    Think in java, notes 
                
    嵌套事务 
                
    java dynamic proxy,动态代理 
                
    埃里克·雷蒙德 
                
    HDU1222 Wolf and Rabbit 
                
    HUT1098 素MM 
                
    HDU1568 Fibonacci 
                
    HDU1501 Zipper DFS+记忆化搜索 
                
                    
          
    • 
          
  • 
            原文地址：https://www.cnblogs.com/lixuebin/p/10814079.html
          
    • 

        

      



      
      

        

        

      

    

  


  

  
Copyright © 2020-2023 
  润新知