Hadoop自带例子学习

用简单例子学习Hadoop，统计字符出现次数。

环境：OS：Centos 6.5 x64 &  Soft：Hadoop 1.2.1

1、创建文件夹，写入字符到文本文件。

[huser@master hadoop-1.2.1]$ mkdir input
[huser@master hadoop-1.2.1]$ echo "hello world" >test1.txt
[huser@master hadoop-1.2.1]$ echo "hello hadoop" >test2.txt

2、写到hdfs中

[huser@master hadoop-1.2.1]$ bin/hadoop fs -put ../input ./in

[huser@master hadoop-1.2.1]$ bin/hadoop fs -ls
Found 1 items
drwxr-xr-x - huser supergroup 0 2014-04-16 19:00 /user/huser/in

[huser@master hadoop-1.2.1]$ bin/hadoop fs -ls ./in/*
-rw-r--r-- 1 huser supergroup 12 2014-04-16 19:00 /user/huser/in/test1.txt
-rw-r--r-- 1 huser supergroup 13 2014-04-16 19:00 /user/huser/in/test2.txt

3、运行自带例子

[huser@master hadoop-1.2.1]$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount in out
14/04/16 19:02:53 INFO input.FileInputFormat: Total input paths to process : 2
14/04/16 19:02:53 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/04/16 19:02:53 WARN snappy.LoadSnappy: Snappy native library not loaded
14/04/16 19:02:54 INFO mapred.JobClient: Running job: job_201404161850_0001
14/04/16 19:02:55 INFO mapred.JobClient: map 0% reduce 0%
14/04/16 19:03:10 INFO mapred.JobClient: map 100% reduce 0%
14/04/16 19:03:19 INFO mapred.JobClient: map 100% reduce 33%
14/04/16 19:03:21 INFO mapred.JobClient: map 100% reduce 100%
14/04/16 19:03:23 INFO mapred.JobClient: Job complete: job_201404161850_0001
14/04/16 19:03:23 INFO mapred.JobClient: Counters: 30
14/04/16 19:03:23 INFO mapred.JobClient: Job Counters 
14/04/16 19:03:23 INFO mapred.JobClient: Launched reduce tasks=1
14/04/16 19:03:23 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=19368
14/04/16 19:03:23 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/04/16 19:03:23 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/04/16 19:03:23 INFO mapred.JobClient: Rack-local map tasks=1
14/04/16 19:03:23 INFO mapred.JobClient: Launched map tasks=2
14/04/16 19:03:23 INFO mapred.JobClient: Data-local map tasks=1
14/04/16 19:03:23 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=11082
14/04/16 19:03:23 INFO mapred.JobClient: File Output Format Counters 
14/04/16 19:03:23 INFO mapred.JobClient: Bytes Written=25
14/04/16 19:03:23 INFO mapred.JobClient: FileSystemCounters
14/04/16 19:03:23 INFO mapred.JobClient: FILE_BYTES_READ=55
14/04/16 19:03:23 INFO mapred.JobClient: HDFS_BYTES_READ=239
14/04/16 19:03:23 INFO mapred.JobClient: FILE_BYTES_WRITTEN=169887
14/04/16 19:03:23 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25
14/04/16 19:03:23 INFO mapred.JobClient: File Input Format Counters 
14/04/16 19:03:23 INFO mapred.JobClient: Bytes Read=25
14/04/16 19:03:23 INFO mapred.JobClient: Map-Reduce Framework
14/04/16 19:03:23 INFO mapred.JobClient: Map output materialized bytes=61
14/04/16 19:03:23 INFO mapred.JobClient: Map input records=2
14/04/16 19:03:23 INFO mapred.JobClient: Reduce shuffle bytes=61
14/04/16 19:03:23 INFO mapred.JobClient: Spilled Records=8
14/04/16 19:03:23 INFO mapred.JobClient: Map output bytes=41
14/04/16 19:03:23 INFO mapred.JobClient: Total committed heap usage (bytes)=415633408
14/04/16 19:03:23 INFO mapred.JobClient: CPU time spent (ms)=4060
14/04/16 19:03:23 INFO mapred.JobClient: Combine input records=4
14/04/16 19:03:23 INFO mapred.JobClient: SPLIT_RAW_BYTES=214
14/04/16 19:03:23 INFO mapred.JobClient: Reduce input records=4
14/04/16 19:03:23 INFO mapred.JobClient: Reduce input groups=3
14/04/16 19:03:23 INFO mapred.JobClient: Combine output records=4
14/04/16 19:03:23 INFO mapred.JobClient: Physical memory (bytes) snapshot=402755584
14/04/16 19:03:23 INFO mapred.JobClient: Reduce output records=3
14/04/16 19:03:23 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2173550592
14/04/16 19:03:23 INFO mapred.JobClient: Map output records=4

4、查看结果

[huser@master hadoop-1.2.1]$ bin/hadoop fs -ls ./out/*
-rw-r--r-- 1 huser supergroup 0 2014-04-16 19:03 /user/huser/out/_SUCCESS
drwxr-xr-x - huser supergroup 0 2014-04-16 19:02 /user/huser/out/_logs/history
-rw-r--r-- 1 huser supergroup 25 2014-04-16 19:03 /user/huser/out/part-r-00000

[huser@master hadoop-1.2.1]$ bin/hadoop fs -cat ./out/part-r-00000
hadoop 1
hello 2
world 1