Spark wordcount开发并提交到集群运行

使用的ide是eclipse

package com.luogankun.spark.base

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

/**
 * 统计字符出现次数
 */
object WordCount {
  def main(args: Array[String]) {
    if (args.length < 1) {
      System.err.println("Usage: <file>")
      System.exit(1)
    }

    val conf = new SparkConf()
    val sc = new SparkContext(conf)
    val line = sc.textFile(args(0))

    line.flatMap(_.split("	")).map((_, 1)).reduceByKey(_+_).collect.foreach(println)
    
    sc.stop
  }
}

导出成jar包。

提交到集群运行脚本: WordCount.sh

#!/bin/bash

set -x

cd $SPARK_HOME/bin

spark-submit 
--name WordCount 
--class com.luogankun.spark.base.WordCount 
--master spark://hadoop000:7077 
--executor-memory 1G 
--total-executor-cores 1 
/home/spark/lib/spark.jar 
hdfs://hadoop000:8020/hello.txt

执行WordCount.sh脚本