hadoop入门学习系列之一hadoop伪分布式模式下安装及运行

一 . 安装JDK

   查询目前系统的JDK： rpm -qa|grep jdk.

   卸载JDK yum -y remove xx

   tar -xzvf jdk-7u80-linux-x64.tar.gz

   配置环境变量： vim /etc/profile

   export JAVA_HOME=/opt/softwares/jdk1.7.0_80

   export PATH=$PATH:$JAVA_HOME/bin

   source /etc/profile

   验证安装 java -version

二 . 配置伪分布式 hadoop

   tar -zxvf hadoop-2.6.5.tar.gz

   修改配置文件

   1. hadoop-env.sh  hadoop要用的环境变量

     export JAVA_HOME=/opt/softwares/jdk1.7.0_80

   2. 修改core-site.xml hadoop core的配置项

   <?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

3. 修改 hdfs-site.xml.hdfs守护进程的配置项

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

4. 修改mapred-site.xml mapreduce的配置项

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

5. 修改yarn-site.xml yarn守护进程的配置项

<?xml version="1.0"?>

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

6. 修改mapred-env.sh和yarn-env.sh文件

export JAVA_HOME=/opt/softwares/jdk1.7.0_80

三 、运行wordcount程序

bin/hdfs namenode -format

sbin/hadoop-daemon.sh start namenode

sbin/hadoop-daemon.sh start datanode

jps

sbin/yarn-daemon.sh start resourcemanager

sbin/yarn-daemon.sh start nodemanager

bin/hdfs dfs -mkdir -p /user/hadoopuser/mapreduce/wordcount/input

bin/hdfs dfs -put wcinput/wc.input /user/hadoopuser/mapreduce/wordcount/input/

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount /user/hadoopuser/mapreduce/wordcount/input/ /user/hadoopuser/mapreduce/wordcount/output