hadoop map（分片）数量确定

之前学习hadoop的时候，一直希望可以调试hadoop源码，可是一直没找到有效的方法，今天在调试矩阵乘法的时候发现了调试的方法，所以在这里记录下来。

1）事情的起因是想在一个Job里设置map的数量(虽然最终的map数量是由分片决定的)，在hadoop1.2.1之前，设置方法是：

job.setNumMapTasks()

不过，hadoop1.2.1没有了这个方法，只保留了设置reduce数量的方法。继续搜索资料，发现有同学提供了另外一种方法，就是使用configuration设置，设置方式如下：

conf.set("mapred.map.tasks",5);//设置5个map

按照上述方法设置之后，还是没有什么效果,控制分片数量的代码如下():

goalSize=totalSize/(numSplits==0？1：numSplits)
//totalSize是输入数据文件的大小，numSplits是用户设置的map数量，就是按照用户自己
//的意愿，每个分片的大小应该是goalSize
minSize=Math.max(job.getLong("mapred.min.split.size",1),minSplitSize)
//hadoop1.2.1中mapred-default.xml文件中mapred.min.split.size=0，所以job.getLong("mapred.min.split.size",1)=0，而minSplitSize是InputSplit中的一个数据成员，在File//Split中值为1.所以minSize=1，其目的就是得到配置中的最小值。
splitSize=Math.max(minSize,Math.min(goalSize,blockSize))
//真正的分片大小就是取按照用户设置的map数量计算出的goalSize和块大小blockSize中最小值(这是为了是分片不会大于一个块大小，有利于本地化计算)，并且又比minSize大的值。

其实，这是hadoop1.2.1之前的生成分片的方式，所以即使设置了map数量也不会有什么实际效果。

2）新版API(hadoop1.2.1)中计算分片的代码如下所示：

 public List<InputSplit> getSplits(JobContext job) throws IOException {
        long minSize = Math.max(this.getFormatMinSplitSize(), getMinSplitSize(job));
        long maxSize = getMaxSplitSize(job);
        ArrayList splits = new ArrayList();
        List files = this.listStatus(job);
        Iterator i$ = files.iterator();

        while(true) {
            while(i$.hasNext()) {
                FileStatus file = (FileStatus)i$.next();
                Path path = file.getPath();
                FileSystem fs = path.getFileSystem(job.getConfiguration());
                long length = file.getLen();
                BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0L, length);
                if(length != 0L && this.isSplitable(job, path)) {
                    long blockSize = file.getBlockSize();
                    long splitSize = this.computeSplitSize(blockSize, minSize, maxSize);

                    long bytesRemaining;
                    for(bytesRemaining = length; (double)bytesRemaining / (double)splitSize > 1.1D; bytesRemaining -= splitSize) {
                        int blkIndex = this.getBlockIndex(blkLocations, length - bytesRemaining);
                        splits.add(new FileSplit(path, length - bytesRemaining, splitSize, blkLocations[blkIndex].getHosts()));
                    }

                    if(bytesRemaining != 0L) {
                        splits.add(new FileSplit(path, length - bytesRemaining, bytesRemaining, blkLocations[blkLocations.length - 1].getHosts()));
                    }
                } else if(length != 0L) {
                    splits.add(new FileSplit(path, 0L, length, blkLocations[0].getHosts()));
                } else {
                    splits.add(new FileSplit(path, 0L, length, new String[0]));
                }
            }

            job.getConfiguration().setLong("mapreduce.input.num.files", (long)files.size());
            LOG.debug("Total # of splits: " + splits.size());
            return splits;
        }
    }

第17行使用computeSplitSize(blockSize,minSize,maxsize)计算分片大小。

a.minSize通过以下方式计算：

 long minSize = Math.max(this.getFormatMinSplitSize(), getMinSplitSize(job))

而getFormatMinSplitSize()：

protected long getFormatMinSplitSize() {
        return 1L;
    }

而getMinSplitSize(job):

 public static long getMinSplitSize(JobContext job) {
        return job.getConfiguration().getLong("mapred.min.split.size", 1L);
    }

没有设置“mapred.min.split.size”的默认值是0。

所以，不设置“mapred.min.split.size”的话，就使用方法的默认值1代替，而“mapred.min.split.size”的默认值是0，所以minSize的值就是1

b.再看maxSize的计算方式：

   long maxSize = getMaxSplitSize(job);

而getMaxSplitSize():

public static long getMaxSplitSize(JobContext context) {
        return context.getConfiguration().getLong("mapred.max.split.size", 9223372036854775807L);
    }

   没有设置"mapred.max.split.size"的话，就使用方法的默认值 9223372036854775807，而"mapred.max.split.size"并没有默认值，所以maxSize= 9223372036854775807；

c.我们已经能够计算出minSize=1，maxSize= 9223372036854775807，接下来计算分片大小：

 long splitSize = this.computeSplitSize(blockSize, minSize, maxSize);

protected long computeSplitSize(long blockSize, long minSize, long maxSize) {
        return Math.max(minSize, Math.min(maxSize, blockSize));
    }

   显然，分片大小是就是maxSize和blockSize的较小值(minSize=1),那么我们就可以通过设置"mapred.max.split.size"来控制map的数量，只要设置值比物理块小就可以了。使用configuration对象的设置方法如下：

conf.set("mapred.max.split.size",2000000)//单位是字节，物理块是16M

3）可以设置map数量的矩阵乘法代码如下所示：

/**
 * Created with IntelliJ IDEA.
 * User: hadoop
 * Date: 16-3-14
 * Time: 下午3:13
 * To change this template use File | Settings | File Templates.
 */
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.util.ReflectionUtils;

public class MutiDoubleInputMatrixProduct {

    public static void initDoubleArrayWritable(int length,DoubleWritable[] doubleArrayWritable){
        for (int i=0;i<length;++i){
            doubleArrayWritable[i]=new DoubleWritable(0.0);
        }
    }

    public static  class MyMapper extends Mapper<IntWritable,DoubleArrayWritable,IntWritable,DoubleArrayWritable>{
        public DoubleArrayWritable map_value=new DoubleArrayWritable();
        public  double[][] leftMatrix=null;/******************************************/
        //public Object obValue=null;
        public DoubleWritable[] arraySum=null;
        public DoubleWritable[] tempColumnArrayDoubleWritable=null;
        public DoubleWritable[] tempRowArrayDoubleWritable=null;
        public double sum=0;
        public double uValue;
        public int leftMatrixRowNum;
        public int leftMatrixColumnNum;
        public void setup(Context context) throws IOException {
            Configuration conf=context.getConfiguration();
            leftMatrixRowNum=conf.getInt("leftMatrixRowNum",10);
            leftMatrixColumnNum=conf.getInt("leftMatrixColumnNum",10);
            leftMatrix=new double[leftMatrixRowNum][leftMatrixColumnNum];
            uValue=(double)(context.getConfiguration().getFloat("u",1.0f));
            tempRowArrayDoubleWritable=new DoubleWritable[leftMatrixColumnNum];
            initDoubleArrayWritable(leftMatrixColumnNum,tempRowArrayDoubleWritable);
            tempColumnArrayDoubleWritable=new DoubleWritable[leftMatrixRowNum];
            initDoubleArrayWritable(leftMatrixRowNum,tempColumnArrayDoubleWritable);
            System.out.println("map setup() start!");
            //URI[] cacheFiles=DistributedCache.getCacheFiles(context.getConfiguration());
            Path[] cacheFiles=DistributedCache.getLocalCacheFiles(conf);
            String localCacheFile="file://"+cacheFiles[0].toString();
            //URI[] cacheFiles=DistributedCache.getCacheFiles(conf);
            //DistributedCache.
            System.out.println("local path is:"+cacheFiles[0].toString());
            // URI[] cacheFiles=DistributedCache.getCacheFiles(context.getConfiguration());
            FileSystem fs =FileSystem.get(URI.create(localCacheFile), conf);
            SequenceFile.Reader reader=null;
            reader=new SequenceFile.Reader(fs,new Path(localCacheFile),conf);
            IntWritable key= (IntWritable)ReflectionUtils.newInstance(reader.getKeyClass(),conf);
            DoubleArrayWritable value= (DoubleArrayWritable)ReflectionUtils.newInstance(reader.getValueClass(),conf);
            //int valueLength=0;
            int rowIndex=0;
            int index;
            while (reader.next(key,value)){
                index=-1;
                for (Writable val:value.get()){ //ArrayWritable类的get方法返回Writable[]数组
                    tempRowArrayDoubleWritable[++index].set(((DoubleWritable)val).get());
                }
                //obValue=value.toArray();
                rowIndex=key.get();
                leftMatrix[rowIndex]=new double[leftMatrixColumnNum];
                //this.leftMatrix=new double[valueLength][Integer.parseInt(context.getConfiguration().get("leftMatrixColumnNum"))];
                for (int i=0;i<leftMatrixColumnNum;++i){
                    //leftMatrix[rowIndex][i]=Double.parseDouble(Array.get(obValue, i).toString());
                    //leftMatrix[rowIndex][i]=Array.getDouble(obValue, i);
                    leftMatrix[rowIndex][i]= tempRowArrayDoubleWritable[i].get();
                }

            }
            arraySum=new DoubleWritable[leftMatrix.length];
            initDoubleArrayWritable(leftMatrix.length,arraySum);
        }
        public void map(IntWritable key,DoubleArrayWritable value,Context context) throws IOException, InterruptedException {
            //obValue=value.toArray();
            InputSplit inputSplit=context.getInputSplit();
            String fileName=((FileSplit)inputSplit).getPath().getName();
            if (fileName.startsWith("FB")) {
                context.write(key,value);
            }
            else{
                int ii=-1;
                for(Writable val:value.get()){
                    tempColumnArrayDoubleWritable[++ii].set(((DoubleWritable)val).get());
                }
                //arraySum=new DoubleWritable[this.leftMatrix.length];
                for (int i=0;i<this.leftMatrix.length;++i){
                    sum=0;
                    for (int j=0;j<this.leftMatrix[0].length;++j){
                        //sum+= this.leftMatrix[i][j]*Double.parseDouble(Array.get(obValue,j).toString())*(double)(context.getConfiguration().getFloat("u",1f));
                        //sum+= this.leftMatrix[i][j]*Array.getDouble(obValue,j)*uValue;
                        sum+= this.leftMatrix[i][j]*tempColumnArrayDoubleWritable[j].get()*uValue;
                    }
                    arraySum[i].set(sum);
                    //arraySum[i].set(sum);
                }
                map_value.set(arraySum);
                context.write(key,map_value);
            }
        }
    }
    public static class MyReducer extends Reducer<IntWritable,DoubleArrayWritable,IntWritable,DoubleArrayWritable>{
        public DoubleWritable[] sum=null;
        // public Object obValue=null;
        public DoubleArrayWritable valueArrayWritable=new DoubleArrayWritable();
        public DoubleWritable[] tempColumnArrayDoubleWritable=null;
        private int leftMatrixRowNum;

        public void setup(Context context){
            //leftMatrixColumnNum=context.getConfiguration().getInt("leftMatrixColumnNum",100);
            leftMatrixRowNum=context.getConfiguration().getInt("leftMatrixRowNum",100);
            sum=new DoubleWritable[leftMatrixRowNum];
            initDoubleArrayWritable(leftMatrixRowNum,sum);
            //tempRowArrayDoubleWritable=new DoubleWritable[leftMatrixColumnNum];
            tempColumnArrayDoubleWritable=new DoubleWritable[leftMatrixRowNum];
            initDoubleArrayWritable(leftMatrixRowNum,tempColumnArrayDoubleWritable);
        }
        //如果矩阵的计算已经在map中完成了，貌似可以不使用reduce，如果不创建reduce类，MR框架仍然会调用一个默认的reduce，只是这个reduce什么也不做
        //但是，不使用reduce的话，map直接写文件，有多少个map就会产生多少个结果文件。这里使用reduce是为了将结果矩阵存储在一个文件中。
        public void reduce(IntWritable key,Iterable<DoubleArrayWritable>value,Context context) throws IOException, InterruptedException {
            //int valueLength=0;
            for(DoubleArrayWritable doubleValue:value){
                int index=-1;
                for (Writable val:doubleValue.get()){
                    tempColumnArrayDoubleWritable[++index].set(((DoubleWritable)val).get());
                }
                //valueLength=Array.getLength(obValue);
                /*
                for (int i=0;i<leftMatrixRowNum;++i){
                    //sum[i]=new DoubleWritable(Double.parseDouble(Array.get(obValue,i).toString())+sum[i].get());
                    //sum[i]=new DoubleWritable(Array.getDouble(obValue,i)+sum[i].get());
                    sum[i].set(tempColumnArrayDoubleWritable[i].get()+sum[i].get());
                }
                */
            }
            //valueArrayWritable.set(sum);
            valueArrayWritable.set(tempColumnArrayDoubleWritable);
            context.write(key,valueArrayWritable);
            /*
            for (int i=0;i<sum.length;++i){
                sum[i].set(0.0);
            }
            */

        }
    }

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        String uri=args[3];
        String outUri=args[4];
        String cachePath=args[2];
        HDFSOperator.deleteDir(outUri);
        Configuration conf=new Configuration();
        DistributedCache.addCacheFile(URI.create(cachePath),conf);//添加分布式缓存
        /**************************************************/
        //FileSystem fs=FileSystem.get(URI.create(uri),conf);
        //fs.delete(new Path(outUri),true);
        /*********************************************************/
        conf.setInt("leftMatrixColumnNum",Integer.parseInt(args[0]));
        conf.setInt("leftMatrixRowNum",Integer.parseInt(args[1]));
        conf.setFloat("u",1.0f);
        //conf.set("mapred.map.tasks",args[5]);
        //int mxSplitSize=Integer.valueOf(args[5])
        conf.set("mapred.max.split.size",args[5]);//hadoop1.2.1中并没有setNumMapTasks方法，只能通过这种方式控制计算分片的大小来控制map数量
        conf.set("mapred.jar","MutiDoubleInputMatrixProduct.jar");
        Job job=new Job(conf,"MatrixProdcut");
        job.setJarByClass(MutiDoubleInputMatrixProduct.class);
        job.setInputFormatClass(SequenceFileInputFormat.class);
        job.setOutputFormatClass(SequenceFileOutputFormat.class);
        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);
        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(DoubleArrayWritable.class);
        job.setOutputKeyClass(IntWritable.class);
        job.setOutputValueClass(DoubleArrayWritable.class);
        FileInputFormat.setInputPaths(job, new Path(uri));
        FileOutputFormat.setOutputPath(job,new Path(outUri));
        System.exit(job.waitForCompletion(true)?0:1);
    }


}
class DoubleArrayWritable extends ArrayWritable {
    public DoubleArrayWritable(){
        super(DoubleWritable.class);
    }
/*
    public String toString(){
        StringBuilder sb=new StringBuilder();
        for (Writable val:get()){
            DoubleWritable doubleWritable=(DoubleWritable)val;
            sb.append(doubleWritable.get());
            sb.append(",");
        }
        sb.deleteCharAt(sb.length()-1);
        return sb.toString();
    }
*/
}

class HDFSOperator{
    public static boolean deleteDir(String dir)throws IOException{
        Configuration conf=new Configuration();
        FileSystem fs =FileSystem.get(conf);
        boolean result=fs.delete(new Path(dir),true);
        System.out.println("sOutput delete");
        fs.close();
        return result;
    }
}

4）接下来说说如何断点调试hadoop源码，这里以计算文件分片的源码为例来说明。

a.首先找到FileInputFormat类，这个类就在hadoop-core-1.2.1.jar中，我们需要将这个jar包添加到工程中，如下所示：

   虽然这是编译之后的类文件，也就是字节码，但是仍然可以像java源码一样，断点调试，这里我们分别在getSplits()方法和computeSplitSize()方法中添加两个断点，然后使用IDEA在本地直接以Debug方式运行我们的MapReduce程序，结果如下所示：

命中断点，并且我们可以查看相关的变量值。