• intellij 调试spark scala 程序 报错


    spark用的是cdh spark-2.0.1
    
    package main.scala
     import org.apache.spark.rdd.RDD
     import org.apache.spark.{SparkConf, SparkContext}
     /* Created by tomtang on 2016/12/16.
      */
    object tomhelloworld {
    
      def FILE_NAME:String = "word_count_results_"
      def main(args: Array[String]) {
    
    //    if (args.length < 1) {
    //      println("Usage:SparkWordCount FileName");
    //      System.exit(1);
    //    }
        val conf = {
          new SparkConf().setMaster("local[*]").setAppName("spark_test")
        }
        val sc = new SparkContext(conf)
        val textFile = sc.textFile("D:\IdeaProjects\first_scala\data\testdata.txt")
        val wordCounts: RDD[(String, Int)] = textFile.flatMap(line => line split " ").map(
          word => (word, 1)).reduceByKey((a, b) => a + b)
        //print the results,for debug use.
        //println("Word Count program running results:")
        //wordCounts.collect().foreach(e => {
        //val (k,v) = e
        //println(k+"="+v)
        //});
        wordCounts.saveAsTextFile(FILE_NAME+System.currentTimeMillis())
        println("Word Count program running results are successfully saved.")
    
        println("Hello World")
      }
    }
    
    报错:
    Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
    at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1410)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:77)
    at sparkTraining.SimpleApp$.main(testScala.scala:9)
    at sparkTraining.SimpleApp.main(testScala.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
    
    参考http://blog.csdn.net/u013887486/article/details/53201900
    因为没有按照文档里面的Scala版本号下载对应的Scala,下载对应的版本2.11.7
    
    发现是scala的版本问题。
    
    
    


    选ivy2.10.4

  • 相关阅读:
    预处理
    关键字 static extern
    linux /proc/cpuinfo 不同处理器显示信息
    sscanf函数简介
    阿里云API接口调用
    AD常用命令以及概念
    开源工具、平台列表
    ELK安装部署
    MySQL8.0初体验
    MySQL案例09:Last_IO_Error: Got fatal error 1236 from master when reading data from binary log
  • 原文地址:https://www.cnblogs.com/tangtianfly/p/6206961.html
Copyright © 2020-2023  润新知