因为Spark与Hadoop是关联的，所以在安装Spark前应该根据已安装的Hadoop版本来选择待安装的Sqark版本，要不然就会报“Server IPC version X cannot communicate with client version Y”的错误。
我安装的Hadoop版本为Hadoop2.4.0(下载)，选择的Spark版本为spark-1.2.0-bin-hadoop2.4.tgz(下载)。要注意的是Spark和Scala存在一定的版本兼容问题，参考我的另一篇博客中记录的问题。
官方文档：http://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts
Spark依赖与Scala，所以还需要预装Scala，我下载的版本为scala-2.11.5.tgz，配置Scala的环境变量：
export SCALA_HOME=/opt/scala/scala-2.11.5
export PATH=$PATH:$SCALA_HOME/bin
修改后使环境变量生效，查看Scala版本：
然后配置Spark的环境变量：
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
配置后使环境变量修改生效。
在 ${SPARK_HOME}/conf 目录下做如下操作：
cp spark-env.sh.template spark-env.sh
修改 spark-env.sh ，在文件最后添加(视具体配置路径而定)：
export SCALA_HOME=/opt/scala/scala-2.11.5 export SPARK_MASTER_IP=127.0.0.1 export SPARK_WORKER_MEMORY=2G export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_75 export HADOOP_HOME=/usr/local/hadoop export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
配置完毕。
启动Spark前，先启动Hadoop：
hadoop@tinylcy:/usr/local/hadoop$ sbin/start-all.sh
然后启动Spark：
hadoop@tinylcy:/opt/spark$ sbin/start-all.sh
切换到Spark的bin目录，进入交互模式：
hadoop@tinylcy:/opt/spark$ bin/spark-shell
测试：
scala> val textFile=sc.textFile("hdfs://localhost:9000/user/hadoop/input/words.txt")
scala> val count=textFile.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)
scala> count.collect()
再举一个例子：
scala> val data=Array(1,2,3,4,5) //产生data data: Array[Int] = Array(1, 2, 3, 4, 5) scala> val distData=sc.parallelize(data) //将data处理成RDD distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:14 scala> distData.reduce(_+_) //在RDD上进行运算，对data里面的元素进行加和 15/07/19 14:37:56 INFO spark.SparkContext: Starting job: reduce at <console>:17 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Got job 0 (reduce at <console>:17) with 4 output partitions (allowLocal=false) 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at <console>:17) 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Parents of final stage: List() 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Missing parents: List() 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Submitting Stage 0 (ParallelCollectionRDD[0] at parallelize at <console>:14), which has no missing parents 15/07/19 14:37:56 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes 15/07/19 14:37:56 INFO storage.MemoryStore: ensureFreeSpace(1184) called with curMem=0, maxMem=278019440 15/07/19 14:37:56 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1184.0 B, free 265.1 MB) 15/07/19 14:37:56 INFO storage.MemoryStore: ensureFreeSpace(912) called with curMem=1184, maxMem=278019440 15/07/19 14:37:56 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 912.0 B, free 265.1 MB) 15/07/19 14:37:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:47649 (size: 912.0 B, free: 265.1 MB) 15/07/19 14:37:56 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0 15/07/19 14:37:56 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:838 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 0 (ParallelCollectionRDD[0] at parallelize at <console>:14) 15/07/19 14:37:56 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1204 bytes) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, PROCESS_LOCAL, 1204 bytes) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, PROCESS_LOCAL, 1204 bytes) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, PROCESS_LOCAL, 1208 bytes) 15/07/19 14:37:56 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 15/07/19 14:37:56 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 15/07/19 14:37:56 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 15/07/19 14:37:56 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 15/07/19 14:37:56 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 727 bytes result sent to driver 15/07/19 14:37:56 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 727 bytes result sent to driver 15/07/19 14:37:56 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 727 bytes result sent to driver 15/07/19 14:37:56 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 727 bytes result sent to driver 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 60 ms on localhost (1/4) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 56 ms on localhost (2/4) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 57 ms on localhost (3/4) 15/07/19 14:37:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 58 ms on localhost (4/4) 15/07/19 14:37:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Stage 0 (reduce at <console>:17) finished in 0.077 s 15/07/19 14:37:56 INFO scheduler.DAGScheduler: Job 0 finished: reduce at <console>:17, took 0.342516 s res0: Int = 15 //得到运算结果 scala>