Flink CEP实例及基础应用

1.flink CEP描述

CEP(Complex Event Processing)就是在无界事件流中检测事件模式，使能够掌握数据中重要的部分。

2.flink CEP编程的四个步骤

1>.输入数据流的创建
2>.模式(Pattern)定义
3>.Pattern应用在时间流上的检测
4>.选取结果

3.常用的个体连续连续模式:

严格连续模式，松散连续，不确定的松散连续。当然还有严格连续的NOT模式和松散连续的NOT模式，这两种并不常用，下面代码举例说明常用的三种模式
flink CEP编程需要导入的lib包

<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-cep_2.11</artifactId>
      <version>${flink.version}</version>
</dependency>

package org.stsffap.cep.monitoring;

import org.apache.flink.cep.CEP;
import org.apache.flink.cep.pattern.Pattern;
import org.apache.flink.cep.pattern.conditions.IterativeCondition;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class MyCEPTest {
    public static void main(String args[]) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStream<String> dataStream = env.fromElements(("a"), ("c"), ("b1"), ("b2"));

        /*---------严格连续模式----------------------*/
        Pattern strictPattern = Pattern.begin("start").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object s, Context<Object> context) {
                return s.toString().equalsIgnoreCase("a");
            }
        }).next("middle").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("b");
            }
        });

        CEP.pattern(dataStream, strictPattern).select(map -> {
            System.out.println("strictPattern:" + map.get("start").toString());
            System.out.println("strictPattern:" + map.get("middle").toString());
            return map;
        }).print();
        /*---------------------------------------------*/

        /*---------松散连续----------------------*/
        Pattern relaxedPattern = Pattern.begin("start").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object s, Context<Object> context) {
                return s.toString().equalsIgnoreCase("a");
            }
        }).followedBy("middle").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("b");
            }
        });

        CEP.pattern(dataStream, relaxedPattern).select(map -> {
            System.out.println("relaxedPattern:" + map.get("start").toString());
            System.out.println("relaxedPattern:" + map.get("middle").toString());
            return map;
        }).print();
        /*---------------------------------------------*/


        /*---------不确定的松散连续----------------------*/
        Pattern nonDeterminPattern = Pattern.begin("start").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object s, Context<Object> context) {
                return s.toString().equalsIgnoreCase("a");
            }
        }).followedByAny("middle").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("b");
            }
        });

        CEP.pattern(dataStream, nonDeterminPattern).select(map -> {
            System.out.println("nonDeterminPattern:" + map.get("start").toString());
            System.out.println("nonDeterminPattern:" + map.get("middle").toString());
            return map;
        }).print();
        /*---------------------------------------------*/

        env.execute("Flink CEP Test");
    }
}

输出结果

nonDeterminPattern:[a]
nonDeterminPattern:[b1]
relaxedPattern:[a]
relaxedPattern:[b1]
nonDeterminPattern:[a]
nonDeterminPattern:[b2]
2> {start=[a], middle=[b2]}
1> {start=[a], middle=[b1]}
1> {start=[a], middle=[b1]}

可以看出严格的连续模式并没有输出结果，因为a和b之间有c，而松散连续输出的结果为(a,b1),不确定的松散连续(a,b1),(a,b2)

4.组合模式举例

上面举例只说明的个体模式较为简单，现在举例说明一个稍微复杂的组合模式举例
a b+c模式:a和b之间是松散连续，b和c之间是严格连续

		DataStream<String> dataStream = env.fromElements(("a"), ("b1"), ("d1"), ("b2"),("d2"),("b3"),("c"));

        //a b+c模式:a和b之间是松散连续，b和c之间是严格连续
        Pattern pattern = Pattern.begin("start").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object s, Context<Object> context) {
                return s.toString().equalsIgnoreCase("a");
            }
        }).followedBy("middle").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("b");
            }
        }).oneOrMore().next("last").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("c");
            }
        });

        CEP.pattern(dataStream, pattern).select(map -> {
            System.out.println("pattern:" + map.get("start").toString());
            System.out.println("pattern:" + map.get("middle").toString());
            System.out.println("pattern:" + map.get("last").toString());
            return map;
        }).print();

输出结果为

pattern:[a]
pattern:[b1, b2, b3]
pattern:[c]
1> {start=[a], middle=[b1, b2, b3], last=[c]}

//a+b c模式:a和b之间是严格连续，b和c之间是松散连续

		DataStream<String> dataStream = env.fromElements(("a"), ("b1"), ("d1"), ("b2"),("d2"),("b3"),("c"));
		//a+b c模式:a和b之间是严格连续，b和c之间是松散连续
        Pattern pattern = Pattern.begin("start").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object s, Context<Object> context) {
                return s.toString().equalsIgnoreCase("a");
            }
        }).next("middle").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("b");
            }
        }).oneOrMore().followedBy("last").where(new IterativeCondition<Object>() {
            @Override
            public boolean filter(Object o, Context<Object> context) {
                return o.toString().contains("c");
            }
        });

        CEP.pattern(dataStream, pattern).select(map -> {
            System.out.println("--------------------------------------");
            System.out.println("pattern:" + map.get("start").toString());
            System.out.println("pattern:" + map.get("middle").toString());
            System.out.println("pattern:" + map.get("last").toString());
            return map;
        }).print();

输出结果为：

--------------------------------------
pattern:[a]
pattern:[b1, b2, b3]
pattern:[c]
--------------------------------------
pattern:[a]
pattern:[b1, b2]
pattern:[c]
--------------------------------------
pattern:[a]
pattern:[b1]
pattern:[c]
3> {start=[a], middle=[b1], last=[c]}
1> {start=[a], middle=[b1, b2, b3], last=[c]}
2> {start=[a], middle=[b1, b2], last=[c]}

5.flink CEP应用场景及总结

flink CEP在实时流数据处理应用中并不仅仅上面介绍的这么简单，还有更多复杂的应用，具体可参照flink官方(https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/libs/cep.html)。