hadoop安装及配置入门篇

声明：
      author： 龚细军
      时间： 17-08-01
      类型： 笔记
      转载时请注明出处及相应链接。
      链接地址： http://www.cnblogs.com/gongxijun/p/5726024.html

本笔记所记录全部基于真实操作所得，所使用hadoop版本为hadoop-2.7.2，使用操作系统为kylin-linux.

默认是：已经安装好了jdk环境.并已经下载好hadoop&解压之后

1. 下载完成hadoo并解压之后

进入到安装目录，我们会看到如下几个文件夹和文件

/hadoop-2.7.2$ ls
bin  include  lib      LICENSE.txt  NOTICE.txt  README.txt  share
etc  input    libexec  logs         output      sbin        wc-in

介绍一下基本情况：

bin目录： hadoop的指令集合存储区，例如 hadoop ,hdfs , yarn,mapred等  这个文件比较重要

我们可以如此使用它们：

/hadoop-2.7.2$ bin/hadoop dfs -cat output/* |more

include目录： C++/C 开发用的头文件 

lib目录： 提供各种库，c/c++开发库

etc目录： 环境配置包，其他的版本采用conf目录替换，进入该目录下会看到

/hadoop-2.7.2/etc/hadoop$ ls | grep .xml
capacity-scheduler.xml
core-site.xml
hadoop-policy.xml
hdfs-site.xml
hdfs-site.xml~
httpfs-site.xml
kms-acls.xml
kms-site.xml
mapred-queues.xml.template
mapred-site.xml.template
ssl-client.xml.example
ssl-server.xml.example
yarn-site.xml

关于如何伪分布式配置 

1.配置文件core.site.xml

  <configuration>
            <property>
            <name>fs.default.name</name>
            <value>hdfs://localhost:9000</value>
            </property>
    </configuration>

2.hdfs.site.xml文件配置

    <configuration>
            <property>
            <name>dfs.replication</name>
            <value>1</value>
            </property>
            <property>
            <name>dfs.name.dir</name>
            <value>/home/gongxijun/HDFS/fileinput</value>
            </property>
            <property>
            <name>dfs.data.dir</name>
            <value>/home/gongxijun/HDFS/fileoutput</value>
            </property>
            <property>
            <name>dfs.permissions</name>
            <value>false</value>
            <description>
            if "true" ,enable permission checking in HDFS. if "false",permission checking is turned off,but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode , owner or group of files or directories.
            </description>
            </property>
            </configuration>

3.配置mapred-site.xml文件，如要将mapred-site.xml.template文件复制一份mapred-site.xml，并对mapred-site.xml进行如下配置

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>
    

之后，启动hadoop，输入./start-all.sh

程序pom.xml文件配置

   <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <artifactId>zookeeper</artifactId>
                    <groupId>org.apache.zookeeper</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jsp-api</artifactId>
                    <groupId>javax.servlet.jsp</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jersey-server</artifactId>
                    <groupId>com.sun.jersey</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>asm</artifactId>
                    <groupId>asm</groupId>
                </exclusion>
            </exclusions>
        </dependency>

运行程序如下：

package com.qunar.mapReduce;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;
import java.util.Scanner;
import java.util.StringTokenizer;

/**
 * *********************************************************
 * <p/>
 * Author:     XiJun.Gong
 * Date:       2016-07-29 14:59
 * Version:    default 1.0.0
 * Class description：
 * <p/>
 * *********************************************************
 */
public class MapReduceDemo {

    public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        @Override
        public void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }

        public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

            @Override
            public void reduce(Text key, Iterable<IntWritable> values, Context context)
                    throws IOException, InterruptedException {
                int sum = 0;
                for (IntWritable val : values) {
                    sum += val.get();
                }

                context.write(key, new IntWritable(sum));
            }
        }


        public static void main(String[] args) throws Exception {

            Configuration configuration = new Configuration();
            Job job = new Job(configuration, "wordCount");
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            job.setMapperClass(Map.class);
            job.setReducerClass(Reduce.class);
            job.setInputFormatClass(TextInputFormat.class);
            job.setOutputFormatClass(TextOutputFormat.class);
            Scanner reader = new Scanner(System.in);
            while (reader.hasNext()) {
                FileInputFormat.addInputPath(job, new Path(reader.next()));
                FileOutputFormat.setOutputPath(job, new Path(reader.next()));
                job.waitForCompletion(true);
            }
        }
    }
}

运行程序：

Connected to the target VM, address: '127.0.0.1:51980', transport: 'socket'
12:41:05.404 [main] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
12:41:05.441 [main] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
12:41:05.442 [main] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[GetGroups], always=false, type=DEFAULT, sampleName=Ops)
12:41:05.444 [main] DEBUG o.a.h.m.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics
12:41:05.871 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty
12:41:05.883 [main] DEBUG org.apache.hadoop.security.Groups -  Creating new Groups object
12:41:05.895 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
12:41:05.896 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
12:41:05.896 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - java.library.path=/home/gongxijun/Qunar/idea-IU-139.1117.1/bin::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
12:41:05.897 [main] WARN  o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12:41:05.900 [main] DEBUG o.a.hadoop.util.PerformanceAdvisory - Falling back to shell based
12:41:05.905 [main] DEBUG o.a.h.s.JniBasedUnixGroupsMappingWithFallback - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
12:41:05.957 [main] DEBUG org.apache.hadoop.util.Shell - setsid exited with exit code 0
12:41:05.957 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
12:41:05.961 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login
12:41:05.962 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login commit
12:41:05.968 [main] DEBUG o.a.h.security.UserGroupInformation - using local user:UnixPrincipal: gongxijun
12:41:05.969 [main] DEBUG o.a.h.security.UserGroupInformation - Using user: "UnixPrincipal: gongxijun" with name gongxijun
12:41:05.969 [main] DEBUG o.a.h.security.UserGroupInformation - User entry: "gongxijun"
12:41:05.970 [main] DEBUG o.a.h.security.UserGroupInformation - UGI loginUser:gongxijun (auth:SIMPLE)

输入命令：

/home/gongxijun/web进阶.txt
/home/gongxijun/a.txt

显示结果：

12:44:36.992 [main] INFO  org.apache.hadoop.mapreduce.Job - Counters: 33
    File System Counters
        FILE: Number of bytes read=6316
        FILE: Number of bytes written=518809
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=84
        Map output records=85
        Map output bytes=1476
        Map output materialized bytes=1652
        Input split bytes=99
        Combine input records=0
        Combine output records=0
        Reduce input groups=82
        Reduce shuffle bytes=1652
        Reduce input records=85
        Reduce output records=82
        Spilled Records=170
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=9
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=459276288
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=1335
    File Output Format Counters 
        Bytes Written=1311

 结果在a.txt文件夹中：

(kafuka卡夫卡)    1
(缺陷：    1
(需要重点学习)    1
---去查看QMQ--message---->broker    1
/    1
1.    2
1.判断线程安全的两个机准：    1
2.    3
3.    1
Apache    1
Cache    1
Client:    1
ConCurrentHashMap    1
Dubbo    1
Executor    1
Futrue/CountDownLatch    1
Guava    1
HTTP:    1
HashMap    1
Hession    1
HttpComponents    1
Java    1
Json    1
Key-Value    1
Kryo(重点)    1
LRU    1
Protobuf    1
QMQ/AMQ/rabbitimq    1
ReadWriterLock    1
ReentrantLock    1
async-http-client    1
c3p0    1
client实现    1
dbpc    1
redis    1
seriialization    1
servlet    1
snchronized    1
spymemcached    1
tomcat-jdbc    1
xmemcached    1
一致性Hash    1
一：    1
三：    1
乐观锁：    1
二：    1
互斥    1
共享数据    1
分布式锁？    1
分布式：    1
前端轮询，后端异步：    1
单例的    1
参数回调    1
可复用资源，创建代价大    1
可扩展性，服务降级，负载均衡，灰度    1
可重入锁    1
可靠性    1
回顾    1
场景：    1
对象池：    1
将对象的状态信息转换为可以存储或传输形式的过程.    1
尽量不要使用本地缓存    1
并发修改    1
序列化：    1
建议：    1
异步调用    1
异步：    1
形成环)    1
性能    1
方式：    1
本地缓存太大，可以使用对象池    1
概念：    1
池化技术    1
消息队列：    1
类型：    1
线程池    1
缓存--本地    1
读写锁：    1
连接池：    1
（分段锁）    1
（推荐使用）    1
，    1