Hadoop全分布模式启动集群

一、 初始化工作：

1. 准备三台机器

hostanme	IP	用途
master	192.168.0.10	namenode
node1	192.168.0.11	datenode
node2	192.168.0.12	datenode

在三台机器上做下面的动作

2. 编辑/etc/hosts

192.168.0.10 master
192.168.0.11 node1
192.168.0.12 node2

3.  创建用户hadoop，及数据存储目录

useradd hadoop
echo hadoop|passwd --stdin hadoop
mkdir /hadoop
mkdir /var/log/hadoop
mkdir /var/run/hadoop
chown -R hadoop:hadoop /hadoop
chown -R hadoop:hadoop /var/log/hadoop /var/run/hadoop

4. 建立节点之间的免密访问hadoop用户

master节点执行

su - hadoop
ssh-keygen
ssh-copy-id hadoop@node1
ssh-copy-id hadoop@node2

同样在node1和node2执行，后面的主机名需要做相应的修改

如果master也作为了datenode，写在了works文件里，则需要把自身设置免密访问

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

5. 安装JAVA运行环境，得到如下效果

[hadoop@node2 ~]$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

[hadoop@node2 ~]$ echo $JAVA_HOME
/usr/java/jdk1.8.0_181

6. 安装需要的依赖

yum -y install pdsh

二、安装并配置Hadoop

在Master节点下执行下面的步骤：

1. 下载hadoop

wget -P /usr/local/src https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

2. 解压到/opt下，并指定属主

在root用户下执行

tar xvf hadoop-3.3.0.tar.gz -C /opt
chown -R hadoop:hadoop /opt/hadoop-3.3.0/

3. 配置hadoop

在hadoop用户下编辑

/opt/hadoop-3.3/etc/hadoop/hadoop-env.sh:

export JAVA_HOME=/usr/java/jdk1.8.0_181
export HADOOP_LOG_DIR=/var/log/hadoop
export HADOOP_PID_DIR=/var/run/hadoop

/opt/hadoop-3.3.0/etc/hadoop/workers:

node1
node2
master

/opt/hadoop-3.3/etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
   </property>
</configuration>

/opt/hadoop-3.3.0/etc/hadoop/mapred-site.xml:

<configuration>
    <property>
        <!-- 历史服务器端地址 -->
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
       <!-- 历史服务器web端地址 -->
       <name>mapreduce.jobhistory.webapp.address</name>
       <value>master:19888</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

/opt/hadoop-3.3.0/etc/hadoop/yarn-site.xml:

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
   </property>
   <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
   </property>
   <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
   </property>
   <property>
        <description>The http address of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
   </property>
   <property>
        <description>The https adddress of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>${yarn.resourcemanager.hostname}:8090</value>
   </property>
   <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
   <property>
        <description>The address of the RM admin interface.</description>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
   </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
        <discription>每个节点可用内存,单位MB,默认8182MB</discription>
   </property>
   <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
   </property>
   <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
    </property>
   <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <!-- 日志聚集功能使能 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>

    <!-- 日志保留时间设置7天 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

4. 复制/opt/hadoop-3.3.0到node1和node2

scp -r /opt/hadoop-3.3.0 root@node1:/opt
scp -r /opt/hadoop-3.3.0 root@node1:/opt

5. 配置hadoop环境变量

在root用户下编辑/etc/profile.d/hadoop.sh

export HADOOP_HOME=/opt/hadoop-3.3.0
export PATH=$PATH:/$HADOOP_HOME/bin

复制到node1和node2

scp /etc/profile.d/hadoop.sh root@node1:/etc/profile.d/
scp /etc/profile.d/hadoop.sh root@node2:/etc/profile.d/

6. 格式化文件系统

在haddop用户下运行

hdfs namenode -format

7. 启动NameNode守护程序和DataNode守护程序

[hadoop@master hadoop-3.3.0]$ ./sbin/start-dfs.sh 
Starting namenodes on [master]
Starting datanodes
Starting secondary namenodes [master]

查看web界面

• NameNode http://192.168.0.10:9870
  

8. 启动ResourceManager守护程序和NodeManager守护程序

[hadoop@master hadoop-3.3.0]$ ./sbin/start-yarn.sh 
Starting resourcemanager
Starting nodemanagers

浏览Web界面以找到ResourceManager；默认情况下，它在以下位置可用：

• ResourceManager- http://192.168.0.10:8088

9. 启动MapReduce JobHistory Server

[hadoop@master hadoop-3.3.0]$ ./bin/mapred --daemon start historyserver

浏览web界面

• historyserver：http://192.168.0.10:19888/

三 在本地运行MapReduce作业

1. 设置执行MapReduce作业所需的HDFS目录：

hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/hadoop

2. 将本地的输入文件上传到HDFS目录

[hadoop@master hadoop-3.3.0]$ hdfs dfs -mkdir input
[hadoop@master hadoop-3.3.0]$ hdfs dfs -put etc/hadoop/*xml input

3. 查看上传到HDFS的文件

[hadoop@master hadoop-3.3.0]$ hdfs dfs -ls /user/hadoop/input
Found 10 items
-rw-r--r--   1 hadoop supergroup       9213 2020-11-19 11:09 /user/hadoop/input/capacity-scheduler.xml
-rw-r--r--   1 hadoop supergroup       1056 2020-11-19 11:09 /user/hadoop/input/core-site.xml
-rw-r--r--   1 hadoop supergroup      11765 2020-11-19 11:09 /user/hadoop/input/hadoop-policy.xml
-rw-r--r--   1 hadoop supergroup        683 2020-11-19 11:09 /user/hadoop/input/hdfs-rbf-site.xml
-rw-r--r--   1 hadoop supergroup       1345 2020-11-19 11:09 /user/hadoop/input/hdfs-site.xml
-rw-r--r--   1 hadoop supergroup        620 2020-11-19 11:09 /user/hadoop/input/httpfs-site.xml
-rw-r--r--   1 hadoop supergroup       3518 2020-11-19 11:09 /user/hadoop/input/kms-acls.xml
-rw-r--r--   1 hadoop supergroup        682 2020-11-19 11:09 /user/hadoop/input/kms-site.xml
-rw-r--r--   1 hadoop supergroup       1388 2020-11-19 11:09 /user/hadoop/input/mapred-site.xml
-rw-r--r--   1 hadoop supergroup       3076 2020-11-19 11:09 /user/hadoop/input/yarn-site.xml

3. 利用示例的jar文件，过滤出input目录中的以dfs开头的单词

[hadoop@master hadoop-3.3.0]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar grep input output 'dfs[a-z.]+'

4. 查看结果

直接在hdfs中查看

[hadoop@master hadoop-3.3.0]$ hdfs dfs -cat output/*
1    dfsadmin
1    dfs.replication
1    dfs.name.dir
1    dfs.data.dir

或者下载到本地查看

 $ bin/hdfs dfs -get output output
 $ cat output/*

四、遇到的问题及解决

问题

node1和node2的datenode已经启动，但是在webui上看不到node1和node2

解决办法：

在node1和node2节点进行下面操作：

需要去修改hdfs-site.xml 中找到配置name和data的2个路径，然后删除标记部分

然后删除实际目录

rm -rf /hadoop/dfs/name

rm -rf /hadoop/dfs/data

修改core-site.xml,删除标记部分

 rm -rf /hadoop/tmp

参考：

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html

https://my.oschina.net/RyenAng/blog/4486155