本文主要记录Spark 在 Eclipse中报 找不到主类的问题。在新建Spark工程,然后add external jars 选择 spark-assembly-1.4.0-hadoop2.6.0.jar 之后,项目工程会出现一个红色的小×,然后发现在 problems中报错:
Description Resource Path Location Type
More than one scala library found in the build path (E:/scalaIDE/eclipse/plugins/org.scala-lang.scala-library_2.12.2.v20170412-161608-VFINAL-21d12e9.jar, D:/spark/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar).At least one has an incompatible version. Please update the project build path so it contains only one compatible scala library. ScalaProject Unknown Scala Classpath Problem
这个错误主要是由于 本地的scala与spark所带的scala冲突导致的。所以我们首先需要将本地的scala删除,右键 scala library container,然后选择build path,remove from build path 即可。此时,可能会发现工程有一个黄色的感叹号。再右键 run as,scala application,结果发现problems中报错:
Description Resource Path Location Type
The version of scala library found in the build path of ScalaProject (2.10.4) is prior to the one provided by scala IDE (2.12.2). Setting a Scala Installation Choice to match. ScalaProject Unknown Scala Version Problem
这里是说,spark 所带的scala 版本与 scala IDE 所提供的版本不一致(spark 的scala 版本更老一些),所以我们需要工程右键,选择properties,然后选择 scala compiler,勾选 use project settings,选择合适的 scala 版本,因为我的spark 带的scala版本比较老,所以这里我选择 latest 2.10 bundle(dynamic),选择apply,OK之后。一切就大功告成了。此时,可以写一个spark的的wordcount,发现就成功运行啦。