1.1 安装及配置

******安装

1、解压并添加软链接

tar -xzvf spark-2.1.0-bin-hadoop2.7.tgz -C /soft/
ln -s /soft/spark-2.1.0-bin-hadoop2.7 /soft/spark

2、添加环境变量

SPARK_HOME=/soft/spark
PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

3、source生效

******此时即可使用 local 模式

进入spark shell

spark-shell

******部署spark集群

1、规划 s101：master，s102-s104：worker

2、s101已装spark，分别在s102-s104装spark

3、s101-s104同步如下配置

　　3.1 、/soft/spark/conf下创建hadoop的core-site.xml和hdfs-site.xml软连接

$>xcall.sh ln -s /soft/hadoop/etc/hadoop/core-site.xml /soft/spark/conf/core-site.xml
$>xcall.sh ln -s /soft/hadoop/etc/hadoop/hdfs-site.xml /soft/spark/conf/hdfs-site.xml

　　3.2、配置JAVA_HOME

$>cd /soft/spark/conf
$>cp spark-env.sh.template spark-env.sh
$>nano spark-env.sh
     ...
     export JAVA_HOME=/soft/jdk
     ...

　　3.3、修改slaves文件

$>cd /soft/spark/conf
$>cp slaves.template slaves
$>nano slaves
s102
s103
s104

4、启动spark集群

//启动hadoop集群
zk start
start-dfs.sh
//启动spark
cd /soft/spark/sbin/
./start-all.sh

[centos@s101 ~]$ xcall.sh jps
==================== s101 jps ===================
2532 NameNode
3012 Master
2855 DFSZKFailoverController
3102 Jps
==================== s102 jps ===================
2438 Worker
2215 DataNode
2503 Jps
2155 QuorumPeerMain
2300 JournalNode
==================== s103 jps ===================
2324 JournalNode
2153 QuorumPeerMain
2219 DataNode
2507 Jps
2444 Worker
==================== s104 jps ===================
2210 DataNode
2149 QuorumPeerMain
2437 Worker
2504 Jps
2314 JournalNode

5、查看spark的webui

　　http://s101:8080/   

6、启动spark shell连接到spark集群

spark-shell --master spark://s101:7077