• Centos下装eclipse测试Hadoop


    (一),安装eclipse

       1,下载eclipse,点这里

       2,将文件上传到Centos7,可以用WinSCP

       3,解压并安装eclipse    

        [root@Master opt]# tar zxvf '/home/s/eclipse-jee-neon-1a-linux-gtk-x86_64.tar.gz' -C/opt  ---------------> 建立文件:[root@Master opt]# mkdir /usr/bin/eclipse     ------------------》添加链接,即快捷方式:[root@Master opt]# ln -s /opt/eclipse/eclipse /usr/bin/eclipse -----------》点击eclipse,即可启动了


    (二),建立Hadoop项目

        1,下载hadoop plugin 2.7.3   链接:http://pan.baidu.com/s/1i5yRyuh 密码:ms91

        2,解压上述jar包插件,放到eclipse中plugins中,并重启eclipse

        2, 在eclipse中加载dfs库,点击Windows 工具栏-------->选择show view如图:

                

        2,打开resource  点击Window ----->Perspective----------->open Perspective  选择resource:

        3,配置连接端口,点击eclipse下放的MapResource Location,点击添加:其中port号按照hdfs-site.xml 和core-site.xml来填写。

        4,上传输入文件:使用hdfs dfs -put /home/file1  /data 即可在eclipse中看到如下:(要确保各个机器的防火墙都关闭,出现异常可以暂时不用关,后面跑下例子就全没了,呵呵)

     


      (三),测试WordCount程序

       1,新建项目:点击new ------------》project ----------->Map Reduce,如图:

       2,给项目配置本地的hadoop文件,圆圈处写本地hadoop的路径:

        

       3,新建个mappert类,写如下代码:

        

    复制代码
     1 package word;
     2 
     3 import java.io.IOException;
     4 import java.util.StringTokenizer;
     5 
     6 import org.apache.hadoop.conf.Configuration;
     7 import org.apache.hadoop.fs.Path;
     8 import org.apache.hadoop.io.IntWritable;
     9 import org.apache.hadoop.io.Text;
    10 import org.apache.hadoop.mapreduce.Job;
    11 import org.apache.hadoop.mapreduce.Mapper;
    12 import org.apache.hadoop.mapreduce.Reducer;
    13 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    14 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    15 import org.apache.hadoop.util.GenericOptionsParser;
    16 
    17 public class mapper {
    18 
    19 public static class TokenizerMapper 
    20 extends Mapper<Object, Text, Text, IntWritable>{
    21 
    22 private final static IntWritable one = new IntWritable(1);
    23 private Text word = new Text();
    24 
    25 public void map(Object key, Text value, Context context
    26 ) throws IOException, InterruptedException {
    27 StringTokenizer itr = new StringTokenizer(value.toString());
    28 while (itr.hasMoreTokens()) {
    29 word.set(itr.nextToken());
    30 context.write(word, one);
    31 }
    32 }
    33 }
    34 
    35 public static class IntSumReducer 
    36 extends Reducer<Text,IntWritable,Text,IntWritable> {
    37 private IntWritable result = new IntWritable();
    38 
    39 public void reduce(Text key, Iterable<IntWritable> values, 
    40 Context context
    41 ) throws IOException, InterruptedException {
    42 int sum = 0;
    43 for (IntWritable val : values) {
    44 sum += val.get();
    45 }
    46 result.set(sum);
    47 context.write(key, result);
    48 }
    49 }
    50 
    51 public static void main(String[] args) throws Exception {
    52 Configuration conf = new Configuration();
    53 
    54 String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    55 if (otherArgs.length != 2) {
    56 System.err.println(otherArgs.length);
    57 System.err.println("Usage: wordcount <in> <out>");
    58 System.exit(2);
    59 }
    60 Job job = new Job(conf, "word count");
    61 job.setJarByClass(mapper.class);
    62 job.setMapperClass(TokenizerMapper.class);
    63 job.setCombinerClass(IntSumReducer.class);
    64 job.setReducerClass(IntSumReducer.class);
    65 job.setOutputKeyClass(Text.class);
    66 job.setOutputValueClass(IntWritable.class);
    67 FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    68 FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
    69 System.out.print("ok");
    70 System.exit(job.waitForCompletion(true) ? 0 : 1);
    71 }
    72 }
    复制代码

    2,点击run as ------------>RunConfigurations ---------->设置input和output文件参数

      

    3,点击run,查看结果

      

      文件的内容:

        


  • 相关阅读:
    Linux系统备份与恢复
    CentOS7修改设置静态IP和DNS
    CentOS系统基础优化16条知识汇总
    CentOS英文提示修改为中文提示的方法
    CentOS修改主机名和网络信息
    CentOS 7系统查看系统版本和机器位数
    Linux下设置SSH Server设置时间链接限制
    查看Linux下系统资源占用常用命令(top、free、uptime)
    查看CentOS系统运行了多久使用uptime命令
    设计模式(七)学习----命令模式
  • 原文地址:https://www.cnblogs.com/xieyulin/p/7061227.html
Copyright © 2020-2023  润新知