MapReduce架构

主从结构

　　主节点：JobTracker（一个）

　　从节点：TaskTrackers（多个）

JobTracker：

　　接收客户提交的计算任务

　　把计算任务分配给TaskTrackers执行

　　监控TaskTracker执行情况

TaskTrackers：

　　执行JobTracker分配的计算任务

---

MapReduce计算模型

　　在Hadoop中，每个MapReduce任务都被初始化为一个Job，每个Job分为两个阶段：Map、Reduce。这两个阶段分别用两个函数表示 ：Map、Reduce

　　Map函数接收一个<key,value>形式的输入，产生同样形式的中间输出。Hadoop将所有相同key的value集合到一起传递给Reduce函数

　　Reduce函数接收一个<key,(list of value)>形式的的呼入，然后对value集合进行处理输出结果。Reduce的输出也是<key,value>的形式

练习：

输入文本

姓名 分数

多个文本，内容行如上述，统计每个人的平均分

Map

package org.zln.scorecount;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;
import java.util.StringTokenizer;

/**
 * Created by sherry on 15-7-12.
 */
public class ScoreMap extends Mapper<LongWritable,Text,Text,IntWritable> {

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();//将纯文本的数据转化为string
        StringTokenizer tokenizer = new StringTokenizer(line,"
");//切割
        while (tokenizer.hasMoreTokens()){
            StringTokenizer tokenizerLine = new StringTokenizer(tokenizer.nextToken());
            String strName = tokenizerLine.nextToken();//姓名
            String strScore = tokenizerLine.nextToken();//成绩

            Text name = new Text(strName);
            int scoreInt = Integer.parseInt(strScore);
            context.write(name,new IntWritable(scoreInt));//输出姓名：成绩

        }
    }
}

Reduce

package org.zln.scorecount;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;
import java.util.Iterator;

/**
 * Created by sherry on 15-7-12.
 */
public class ScoreReduce extends Reducer<Text,IntWritable,Text,IntWritable> {
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int sum = 0;
        int count = 0;
        Iterator<IntWritable> intWritableIterator = values.iterator();
        while (intWritableIterator.hasNext()){
            sum += intWritableIterator.next().get();//总分
            count++;//平均分
        }
        int avg = sum/count;
        context.write(key,new IntWritable(avg));
    }
}

Main

package org.zln.scorecount;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 * Created by sherry on 15-7-12.
 */
public class ScoreMain extends Configured implements Tool{
    public int run(String[] args) throws Exception {
        Job job = new Job(getConf());
        job.setJarByClass(ScoreMain.class);
        job.setJobName("ScoreCount");


        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        job.setMapperClass(ScoreMap.class);
        job.setReducerClass(ScoreReduce.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        boolean success = job.waitForCompletion(true);
        return success?0:1;
    }

    //统计平均分
    public static void main(String[] args) throws Exception {
        int ret = ToolRunner.run(new ScoreMain(), args);
        System.exit(ret);
    }
}

---

---

我们的Map与Reduce都继承了父类，并复写了map或reduce方法

父类中 还有 三个方法未作处理

setup：启动map/reduce后首先调用

cleanup：最后调用

run：每次调用的时候都会执行