阿里云服务器编译安装Hadoop 2.7.4 伪分布式环境

一、系统环境

　　采用的是阿里云的Centos 7 64 bit 操作系统

二、安装方式

　　有两种安装方式：一种是安装已经编译好的Hadoop 安装压缩包；一种是自己编译安装。本次选择编译安装，因为我们从官网下载的Hadoop安装压缩包（hadoop-2.7.4.tar.gz）是32系统下的，而我们的系统是64位的，最好我们自己编译64位的安装包进行安装。当然也可以安装别人编译好的64位安装包。

三、安装依赖

　　在编译Hadoop之前需要安装一些依赖软件，主要是Jdk、ant、findbugs和protobuf等，加上从官网下载的Hadoop源码包共需要5个安装包：

  

3.1 建立环境所需的文件夹

　　1）建立三个文件夹：

• 　　tool：存放我们的安装包；
• 　　softwores：软件安装路径
• 　　data：数据存放

　　博主是在/home下建立的三个文件夹

[root@hadoop home]# mkdir tools
[root@hadoop home]# mkdir softwares
[root@hadoop home]# mkdir data
[root@hadoop home]# ll
total 12
drwxr-xr-x 2 root root 4096 Dec 11 11:22 data
drwxr-xr-x 9 root root 4096 Dec 11 11:05 softwares
drwxr-xr-x 2 root root 4096 Dec 10 22:14 tools
[root@hadoop home]# 

　　2）安装 jdk、ant 和findbugs，将上图中对应的安装包上传到tools文件夹中，并进行解压

[root@hadoop tools]# tar -zxvf apache-ant-1.9.4-bin.tar.gz -C /home/softwares/
[root@hadoop tools]# tar -zxvf findbugs-1.3.9.tar.gz -C /home/softwares/
[root@hadoop tools]# tar -zxvf jdk-7u80-linux-x64.tar.gz -C /home/softwares/

　　　  配置环境变量（编辑/etc/profile文件）

export JAVA_HOME=/home/softwares/jdk1.7.0_80
export ANT_HOME=/home/softwares/apache-ant-1.9.4
export FINDBUGS_HOME=/home/softwares/findbugs-1.3.9
export PATH=$PATH:$FINDBUGS_HOME/bin:$ANT_HOME/bin:$JAVA_HOME/bin

　　      使配置生效并进行验证

[root@hadoop tools]# source /etc/profile
[root@hadoop tools]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
[root@hadoop tools]# ant -version
Apache Ant(TM) version 1.9.4 compiled on April 29 2014
[root@hadoop tools]# findbugs -version
1.3.9
[root@hadoop tools]# 

　　3）安装其他依赖软件（需联网）

[root@hadoop tools]# yum -y install maven svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

　　4）安装protobuf，将上图中对应的安装包上传到tools文件夹中，并进行解压

[root@hadoop tools]# tar -zxvf protobuf-2.5.0.tar.gz -C /home/softwares/

　　    protobuf安装并验证

[root@hadoop tools]# cd /home/softwares/protobuf-2.5.0/
[root@hadoop protobuf-2.5.0]# ./configure
[root@hadoop protobuf-2.5.0]# make && make install
[root@hadoop protobuf-2.5.0]# protoc --version
libprotoc 2.5.0

 　　至此，所有的依赖已经安装完成。

 四、编译Hadoop源码

　　4.1 进行编译

[root@hadoop protobuf-2.5.0]# cd /home/softwares/hadoop-2.7.4
[root@hadoop hadoop-2.7.4]# mvn package -Pdist,native -DskipTests -Dtar

　　接下来是漫长的等待过程 .... .... ..... .... .... ... 

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  3.533 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.023 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.679 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.275 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.875 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.856 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.340 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.534 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  5.398 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [03:02 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 11.653 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 24.501 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.112 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [07:28 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 41.608 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 10.673 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  7.225 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.057 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.110 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [03:36 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 45.418 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.164 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 12.942 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 19.200 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  3.315 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  7.855 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 24.347 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  6.439 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  6.393 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  3.445 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.075 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.304 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  2.026 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.155 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  7.255 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 11.871 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.254 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 26.029 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 25.002 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.792 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.797 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  5.143 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  6.771 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.837 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.513 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  6.842 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  4.355 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 14.910 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  2.844 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  6.931 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.937 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.499 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.268 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.739 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  5.793 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.444 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.258 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 47.689 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 19.524 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.305 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  5.581 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 25.708 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.281 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:53 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:49 min
[INFO] Finished at: 2015-12-11T20:29:45+08:00
[INFO] Final Memory: 110M/493M
[INFO] ------------------------------------------------------------------------

　　安装完成后如上所示（我自己的没有了，这个是网上找的，我自己用了1个多小时）

　　编译好的文件在 /home/softwares/hadoop-2.7.4-src/hadoop-dist/target/

[root@hadoop target]# pwd
/home/softwares/hadoop-2.7.4-src/hadoop-dist/target
[root@hadoop target]# ll
total 591308
drwxr-xr-x 2 root root      4096 Dec 11 10:59 antrun
drwxr-xr-x 3 root root      4096 Dec 11 10:59 classes
-rw-r--r-- 1 root root      1872 Dec 11 10:59 dist-layout-stitching.sh
-rw-r--r-- 1 root root       645 Dec 11 10:59 dist-tar-stitching.sh
drwxr-xr-x 9 root root      4096 Dec 11 10:59 hadoop-2.7.4
-rw-r--r-- 1 root root 201372886 Dec 11 10:59 hadoop-2.7.4.tar.gz
-rw-r--r-- 1 root root     26520 Dec 11 10:59 hadoop-dist-2.7.4.jar
-rw-r--r-- 1 root root 403999148 Dec 11 11:00 hadoop-dist-2.7.4-javadoc.jar
-rw-r--r-- 1 root root     24048 Dec 11 10:59 hadoop-dist-2.7.4-sources.jar
-rw-r--r-- 1 root root     24048 Dec 11 10:59 hadoop-dist-2.7.4-test-sources.jar
drwxr-xr-x 2 root root      4096 Dec 11 10:59 javadoc-bundle-options
drwxr-xr-x 2 root root      4096 Dec 11 10:59 maven-archiver
drwxr-xr-x 3 root root      4096 Dec 11 10:59 maven-shared-archive-resources
drwxr-xr-x 3 root root      4096 Dec 11 10:59 test-classes
drwxr-xr-x 2 root root      4096 Dec 11 10:59 test-dir
[root@hadoop target]# 

 　　至此已经编译好了hadoop。

 五、Hadoop 安装和配置启动（未完待续...）

 六、踩过的坑

 七、参考的博客

　　http://blog.csdn.net/miracle_yan/article/details/78288552　

　    http://blog.csdn.net/young_kim1/article/details/50269501