数据分析之mapreduce

Mapreduce程序分析

一．今日任务

向Hadoop平台提交日志文件dat0203.log，并使用streaming和MapReduce机制编制程序，统计日志文件dat0203.log的数据中一共包含多少部电影？本题的赛前抽取参数是dat0203.log文件，请参赛学生用hdfs命令查看输出的结果，截屏保存于图片ans0203.jpg，并用hdfs命令把输出文件传输到本地，修改文件名为ans0203.txt 

二．任务源码

Mapper

package sss;

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

public class SMapper extends Mapper <LongWritable, Text, Text, LongWritable>{

@Override

protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, LongWritable>.Context context)

throws IOException, InterruptedException {

// TODO Auto-generated method stub

String line = value.toString();

String s[] = line.split(";");

int num = 0;

for(String str : s) {

if(s.equals("")) {

continue;

}

num ++;

}

context.write(new Text("movie"), new LongWritable(num));

}

}

package sss;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class Submitter {

public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

Configuration conf = new Configuration();

FileSystem fs = FileSystem.get(conf);

if(fs.exists(new Path(args[1]))) {

fs.delete(new Path(args[1]), true);

}

Job job = Job.getInstance();

job.setJarByClass(Submitter.class);

job.setMapperClass(SMapper.class);

job.setMapOutputKeyClass(Text.class);

job.setMapOutputValueClass(LongWritable.class);

FileInputFormat.setInputPaths(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

boolean b = job.waitForCompletion(true);

System.out.println(b);

}

}

程序运行结果

 

三．遇到问题

四．解决方案