第90讲：基于Scala的Actor之上的分布式并发消息驱动框架Akka初体验
akka在业界使用非常广泛
spark背后就是由akka驱动的
要写消息驱动的编程模型都首推akka
下面将用30讲讲解akka
本讲主要讲两部分内容：
1.akka的重大意义
2.akka与scala的actor
Spark源码中使用akka使用鉴赏：
在spark中有200行左右代码封装了akka的使用
spark是分布式的计算框架，有master和slave主从节点通信时都是使用akka。
客户端提交程序时也是使用akka。所以如果要掌握spark必须要理解和掌握akka。
第91讲：Akka第一个案例动手实战架构设计
The quick brown fox tried to jump over the lazy dog and fell on the dog
A dog is a man's best friend
Map Task -> Reduce Task --|
|-> Aggregate Task
Map Task -> Reduce Task --|
Master Actor
|
--------------------|---------------------
| | |
Map Actor Reduce Actor Aggregate Actor
Master Actor给MapActor发一个字符串
Map Actor根据规则对单词计数，计数完成后把结果传递给MasterActor
MasterActor把MapData以消息发给reduceActor，
reduceActor会reduceByKey,把相同单词（key相同）计数相加。
计算完后再把数据传回给MasterActor。
如果有多条字符串就会有多组reduce结果。
MasterActor再把结果发给AggregateActor，进行最后统计
MasterActor要获得结果需要给AggregateActor发一个空消息，
AggregateActor收到消息就会把所有统计结果发给MasterActor
这就是mapReduce计算模型。
与hadoop的mapreduce不同的是这是基于actor的。
MapActor对map产生的结果进行本地化统计，
AggregateActor才相当于hadoop的reducer。
后面先通过java使用akka。
第92讲：Akka第一个案例动手实战开发环境的搭建
第93讲：Akka第一个案例动手实战开发消息实体类
MapData:
package akka.dt.app.java.messages;
import java.util.List;
/**
* sunrunzhi
* 2018/11/12 9:46
* 用来让MapActor处理数据后存储在MapData实体中
* 然后方便将结果交给ReduceActor
*/
public class MapData {
private List<WordCount> dataList;
public List<WordCount> getDataList(){return dataList;}
public MapData(List<WordCount> dataList){
this.dataList=dataList;
}
}
ReduceData:
package akka.dt.app.java.messages;
import java.util.HashMap;
/**
* sunrunzhi
* 2018/11/12 13:35
*/
public class ReduceData {
private HashMap<String,Integer> reduceDataList;
public HashMap<String,Integer> getReduceDataList(){
return reduceDataList;
}
public ReduceData(HashMap<String,Integer> reduceDataList){
this.reduceDataList=reduceDataList;
}
}
Result:
package akka.dt.app.java.messages;
/**
* sunrunzhi
* 2018/11/12 9:58
*/
public class Result {
/*传入的字符串先交给MapActor进行切分，然后交给ReduceActor进行本地统计，
最后交给AggregateActor进行全局的统计，
想要获得这个结果，通过MasterActor发一个消息Result，Result本身为空，不需要有任何内容。
这个消息交给MasterActor，MasterActor收到消息时，如果消息是result类型的话转过来会告诉AggregateActor,
再转发给AggregateActor。*/
}
WordCount:
package akka.dt.app.java.messages;
/**
* sunrunzhi
* 2018/11/12 9:52
* WordCount-javaBean
*/
public class WordCount {
private String word;
private Integer count;
public WordCount(String inWord,Integer inCount){
word=inWord;
count=inCount;
}
public String getWord(){return word;}
public Integer getCount(){return count;}
}
HelloAkka:
package akka.dt.app.java.messages;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import akka.actor.Props;
import akka.dt.app.java.actors.MasterActor;
/**
* sunrunzhi
* 2018/11/9 20:14
*/
public class HelloAkka {
public static void main(String[] args)throws Exception{
ActorSystem _system=ActorSystem.create("HelloAkka");
ActorRef master=_system.actorOf(new Props(MasterActor.class),"master");
master.tell("Hi,My name is Rocky. I'm so so so so happy to me here.");
master.tell("Today,I'm going to read a news article for so you.");
master.tell("I hope I hope you'll like it.");
Thread.sleep(500);
master.tell(new Result());
Thread.sleep(500);
_system.shutdown();
}
}
第94讲：Akka第一个案例动手实战MapActor、ReduceActor、AggregateActor代码详解
AggregateActor:
package akka.dt.app.java.actors;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.ReduceData;
import akka.dt.app.java.messages.Result;
import java.util.HashMap;
import java.util.Map;
/**
* sunrunzhi
* 2018/11/9 20:18
*/
public class AggregateActor extends UntypedActor{
private Map<String,Integer> finalReducedMap=new HashMap<String, Integer>();
@Override
public void onReceive(Object message)throws Exception{
//AggregateActor收到消息有两种，一种是ReduceData类型，一种是Result类型
if(message instanceof ReduceData){
ReduceData reduceData=(ReduceData)message;
aggregateInMemoryReduce(reduceData.getReduceDataList());
}else if(message instanceof Result){
System.out.println(finalReducedMap.toString());
}else{
unhandled(message);
}
}
private void aggregateInMemoryReduce(Map<String,Integer> reduceList){
Integer count=null;
for (String key:reduceList.keySet()){
if(finalReducedMap.containsKey(key)){
count=reduceList.get(key)+finalReducedMap.get(key);
finalReducedMap.put(key,count);
}else{
finalReducedMap.put(key,reduceList.get(key));
}
}
}
}
MapActor:
package akka.dt.app.java.actors;
import akka.actor.ActorRef;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.MapData;
import akka.dt.app.java.messages.WordCount;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.StringTokenizer;
/**
* sunrunzhi
* 2018/11/9 20:17
*/
public class MapActor extends UntypedActor {
String[] STOP_WORDS = {"a", "is"};
private ActorRef reduceActor = null;
private List<String> STOP_WORDS_LIST = Arrays.asList(STOP_WORDS);
public MapActor(ActorRef inReduceActor) {
reduceActor = inReduceActor;
}
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
String work = (String) message;
//map the words in the sentence
MapData data = evaluateExpression(work);//产生MapData实体
// send the result to ReduceActor
reduceActor.tell(data);
} else {
unhandled(message);
}
}
private MapData evaluateExpression(String line) {
List<WordCount> dataList = new ArrayList<WordCount>();
StringTokenizer parser = new StringTokenizer(line);
while (parser.hasMoreTokens()){
String word =parser.nextToken().toLowerCase();
if (!STOP_WORDS_LIST.contains(word)) {
dataList.add(new WordCount(word, Integer.valueOf(1)));
}
}
return new MapData(dataList);
}
}
ReduceActor:
package akka.dt.app.java.actors;
import akka.actor.ActorRef;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.MapData;
import akka.dt.app.java.messages.ReduceData;
import akka.dt.app.java.messages.WordCount;
import com.sun.javafx.collections.MappingChange;
import java.util.HashMap;
import java.util.List;
/**
* sunrunzhi
* 2018/11/9 20:17
*/
public class ReduceActor extends UntypedActor{
private ActorRef aggregateActor =null;
public ReduceActor(ActorRef inAggregateActor) {
aggregateActor=inAggregateActor;
}
@Override
public void onReceive(Object message) throws Exception{
//AggregateActor收到的消息有两种，一种是ReduceData类型，一种是Result类型
if(message instanceof MapData){
MapData mapData =(MapData)message;
//reduce the incoming data
ReduceData reduceData=reduce(mapData.getDataList());
//forward the result to aggregate actor
aggregateActor.tell(reduceData);
}else {
unhandled(message);
}
}
private ReduceData reduce(List<WordCount> dataList){
HashMap<String,Integer> reducedMap=new HashMap<String, Integer>();
for(WordCount wordCount:dataList){
if(reducedMap.containsKey(wordCount.getWord())){
Integer value=(Integer)reducedMap.get(wordCount.getWord());
value++;
reducedMap.put(wordCount.getWord(),value);
}else{
reducedMap.put(wordCount.getWord(),Integer.valueOf(1));
}
}
return new ReduceData(reducedMap);
}
}
MasterActor:
package akka.dt.app.java.actors;
import akka.actor.*;
import akka.dt.app.java.messages.Result;
import java.util.Arrays;
import java.util.List;
/**
* sunrunzhi
* 2018/11/9 20:18
*/
public class MasterActor extends UntypedActor {
private ActorRef aggregaterActor = getContext().actorOf(
new Props(AggregateActor.class), "aggregate");
private ActorRef reduceActor=getContext().actorOf(
new Props(new UntypedActorFactory() {
public UntypedActor create() {
return new ReduceActor(aggregaterActor);
}
}),"reduce");
private ActorRef mapActor=getContext().actorOf(
new Props(new UntypedActorFactory(){
public UntypedActor create(){
return new MapActor(reduceActor);
}
}),"map");
@Override
public void onReceive(Object message) throws Exception{
if (message instanceof String) {
mapActor.tell(message);
} else if (message instanceof Result) {
aggregaterActor.tell(message);
} else{
unhandled(message);
}
}
}
第95讲：Akka第一个案例动手实战MasterActor代码详解
第96讲：Akka第一个案例动手实战main方法实现中ActorSystem等代码详解
第97讲：使用SBT开发Akka第一个案例环境搭建详解
SBT import失败：
一步一个坑。凑人~~~
之前调整的项目有报莫名其妙的BUG。
第一个BUG。
解决方案：标识红色处打上对号。
总有些奇奇怪怪的BUG，莫名其妙的又没有了........