Flume+Kafka+Storm整合

1. 需求：

有一个客户端Client可以产生日志信息，我们需要通过Flume获取日志信息，再把该日志信息放入到Kafka的一个Topic：flume-to-kafka

再由Storm读取该topic：flume-to-kafka，进行日志分析处理（这里我们做的逻辑处理为filter，即过滤日志信息），处理完日志信息后，再由Storm把处理好的日志信息放入到Kafka的另一个topic：storm-to-kafka

2.组件分布情况

我总共搭建了3个节点node1,node2,node3

Zookeeper安装在node1,node2,nod3

Flume安装在node2

Kafka安装在node1,node2,node3

Storm安装在node1,node2,node3

3.JDK安装

--在node1， node2， node3上面安装jdk
--install JDK   -- http://blog.51cto.com/vvxyz/1642258（LInux安装jdk的三种方法）
--解压安装
rpm -ivh your-package.rpm

--修改环境变量
vi /etc/profile

JAVA_HOME=/usr/java/jdk1.7.0_67
JRE_HOME=/usr/java/jdk1.7.0_67/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

：wq

--使配置有效
source /etc/profile

4.Zookeeper的安装

---============================
--解压zookeeper压缩包并安装
tar -zxvf zookeeper-3.4.6.tar.gz 

--创建zookeeper的软链
ln -sf /root/zookeeper-3.4.6 /home/zk

--配置zookeeper
cd /home/zk/conf/

--把下面的zoo_sample.cfg文件重新命名
cp zoo_sample.cfg zoo.cfg

--修改zoo.cfg配置文件
vi zoo.cfg

--设置zookeeper的文件存放目录
--找到dataDir=/tmp/zookeeper，并设置为下面值
dataDir=/opt/zookeeper

--设置zookeeper集群
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

:wq


--创建/opt/zookeeper目录
mkdir /opt/zookeeper

--进入/opt/zookeeper目录
cd /opt/zookeeper

--创建一个文件myid
vi myid

--输入1
1
：wq
--以此类推，在node2，node3，值分别是2， 3

--拷贝zookeeper目录到node2， node3的/opt/目录下面
cd ..
scp -r zookeeper/ root@node2:/opt/
scp -r zookeeper/ root@node3:/opt/

--分别进入到node2， node3里面，修改/opt/zookeeper/myid，值分别是2， 3


--作为以上配置，把node1里面的zookeeper拷贝到node2， node3上面。
scp -r zookeeper-3.4.6 root@node2:~/
scp -r zookeeper-3.4.6 root@node3:~/

--分别进入到node2， node3里面，创建软链
ln -sf /root/zookeeper-3.4.6/ /home/zk


--配置zookeeper环境变量
cd /home/zk/bin

--修改/etc/profile文件，把zookeeper的bin目录路径添加进去
vi /etc/profile
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:/home/zk/bin

--让配置文件生效
source /etc/profile

--分别进入到node2， node3里面，修改/etc/profile文件，把zookeeper的bin目录路径添加进去


--作为环境变量配置，就可以启动zookeeper了。
--分别在node1， node2， node3上面启动zookeeper
zkServer.sh start


--测试是否启动成功
jps
--观察是否有QuorumPeerMain进程

5.Flume的安装

---------------------------------------------------
--安装Flume
--把安装包上传到node2上面
cd
tar -zxvf apache-flume-1.6.0-bin.tar.gz 

--创建软链
ln -s /root/apache-flume-1.6.0-bin /home/flume

--配置flume
cd /root/apache-flume-1.6.0-bin/conf

cp flume-env.sh.template flume-env.sh

vi flume-env.sh

--配置JDK
export JAVA_HOME=/usr/java/jdk1.7.0_67

:wq


--加入系统变量
vi /etc/profile

export FLUME_HOME=/root/apache-flume-1.6.0-bin
export PATH=$PATH:$FLUME_HOME/bin

:wq

source /etc/profile

--验证是否安装成功
flume-ng version

flume-ng version
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f

6.Kafka的安装

---------------------------------------------------
--kafka安装
--在node1， node2, node3上面搭建kafka
--先进入node1
mkdir /root/kafka

--解压
tar -zxvf kafka_2.10-0.8.2.2.tgz

--创建软链
ln -s /root/kafka/kafka_2.10-0.8.2.2 /home/kafka-2.10

--配置
cd /root/kafka/kafka_2.10-0.8.2.2/config

vi server.properties

--node1=0, node2=1,node2=2
broker.id=0
log.dirs=/opt/kafka-log
zookeeper.connect=node1:2181,node2:2181,node3:2181

:wq


--为了启动方便
cd /root/kafka/kafka_2.10-0.8.2.2
vi start-kafka.sh

nohup bin/kafka-server-start.sh  config/server.properties > kafka.log 2>&1 &

:wq

chmod 755 start-kafka.sh



--配置好以后
--分发到node2，node3
cd /root/kafka/
scp -r kafka_2.10-0.8.2.2/ root@node2:/root/kafka
scp -r kafka_2.10-0.8.2.2/ root@node3:/root/kafka

--进入到node2
cd /root/kafka/kafka_2.10-0.8.2.2/config

vi server.properties

--node1=0, node2=1,node2=2
broker.id=1

:wq


--进入到node3
cd /root/kafka/kafka_2.10-0.8.2.2/config

vi server.properties

--node1=0, node2=1,node2=2
broker.id=2

:wq

--启动kafka
./zkServer.sh start

--分别进入node1,node2,node3
cd /root/kafka/kafka_2.10-0.8.2.2
./start-kafka.sh


--检查是否启动
jps
查看是否有Kafka进程

7.Storm的安装

------------
--Storm分布式安装
--部署到node1，node2，node3节点上
--进入node1
cd /root/apache-storm-0.10.0/conf

vi storm.yaml


--配置如下
# storm.zookeeper.servers:
     - "node1"
     - "node2"
     - "node3"
#  
nimbus.host: "node1"
storm.local.dir: "/opt/storm"
supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703

:wq

--从node1分发到node2，node3
scp -r apache-storm-0.10.0 root@node2:/
scp -r apache-storm-0.10.0 root@node3:/


--分别进入node2，node3创建软链
ln -r /root/apache-storm-0.10.0 /home/storm-0.10


--分别进入node1,node2，node3快捷启动
cd /root/apache-storm-0.10.0

vi start-storm.sh

nohup bin/storm nimbus >> logs/numbus.out 2>&1 &
nohup bin/storm supervisor >> logs/supervisor.out 2>&1 &


--node1上面配置，node2，node3上面不需要UI
nohup bin/storm ui >> logs/ui.out 2>&1 &
nohup bin/storm drpc >> logs/drpc.out 2>&1 &


:wq

--分别进入node1,node2，node3快捷stop-storm

vi stop-storm.sh

--node1上面配置，node2，node3上面不需要UI
kill -9 `jps | grep core | awk '{print $1}'`


kill -9 `jps | grep supervisor | awk '{print $1}'`
kill -9 `jps | grep nimbus | awk '{print $1}'`
kill -9 `jps | grep worker | awk '{print $1}'`
kill -9 `jps | grep LogWriter | awk '{print $1}'`

:wq


chmod 755 start-storm.sh
chmod 755 stop-storm.sh



--启动Zookeeper服务
--在node1，node2，node3上面启动
zkServer.sh start

--在node1，node2，node3上面启动Storm
cd /root/apache-storm-0.10.0
./start-storm.sh


--上传storm_wc.jar 文件
./storm jar /root/storm_wc.jar storm.wordcount.Test wordcount


------------
Storm DRPC 配置
--进入node1
cd /root/apache-storm-0.10.0/conf

vi storm.yaml

drpc.servers:     
     - "node1"

:wq

--从node1，分发到node2，node3
cd /root/apache-storm-0.10.0/conf/
scp -r root@node2:/root/apache-storm-0.10.0/conf
scp -r root@node3:/root/apache-storm-0.10.0/conf


--配置完，进入node1，node2，node3
cd /root/apache-storm-0.10.0
./start-storm.sh &

8.Flume+Kafka+Storm整合

8.1.配置Flume

--从node2
cd flumedir

vi flume_to_kafka

--node2配置如下
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = node2
a1.sources.r1.port = 41414

# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = flume-to-kafka
a1.sinks.k1.brokerList = node1:9092,node2:9092,node3:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

:wq

8.2.启动Zookeeper

--启动Zookeeper,在node1，node2，node3
--关闭防火墙
service iptables stop

--启动Zookeeper
zkServer.sh start

8.3.启动Kfaka

--启动kafka
--分别进入node1,node2,node3
cd /root/kafka/kafka_2.10-0.8.2.2
./start-kafka.sh

8.4.启动Flume

--进入node2，启动
cd /root/flumedir

flume-ng agent -n a1 -c conf -f flume_to_kafka -Dflume.root.logger=DEBUG,console

8.4.启动客户端Client

启动客户端产生日志信息。

大家可以参考RPC clients - Avro and Thrift的代码

import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.api.RpcClient;
import org.apache.flume.api.RpcClientFactory;
import org.apache.flume.event.EventBuilder;
import java.nio.charset.Charset;

public class MyApp {
    public static void main(String[] args) {
        MyRpcClientFacade1 client = new MyRpcClientFacade1();
        // Initialize client with the remote Flume agent's host and port
        client.init("node2", 41414);

        // Send 10 events to the remote Flume agent. That agent should be
        // configured to listen with an AvroSource.
        String sampleData = "Hello ERROR ! ------ Test";
        for (int i = 500; i < 505; i++) {
            client.sendDataToFlume(sampleData + " " + i);
            System.out.println(sampleData + " " + i);
        }

        client.cleanUp();
    }
}

class MyRpcClientFacade1 {
    private RpcClient client;
    private String hostname;
    private int port;

    public void init(String hostname, int port) {
        // Setup the RPC connection
        this.hostname = hostname;
        this.port = port;
        this.client = RpcClientFactory.getDefaultInstance(hostname, port);
        // Use the following method to create a thrift client (instead of the
        // above line):
        // this.client = RpcClientFactory.getThriftInstance(hostname, port);
    }

    public void sendDataToFlume(String data) {
        // Create a Flume Event object that encapsulates the sample data
        Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));

        // Send the event
        try {
            client.append(event);
        } catch (EventDeliveryException e) {
            // clean up and recreate the client
            client.close();
            client = null;
            client = RpcClientFactory.getDefaultInstance(hostname, port);
            // Use the following method to create a thrift client (instead of
            // the above line):
            // this.client = RpcClientFactory.getThriftInstance(hostname, port);
        }
    }

    public void cleanUp() {
        // Close the RPC connection
        client.close();
    }
}

在eclipse控制台输出的结果是：

[ WARN ] - [ org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:505) ] Invalid value for batchSize: 0; Using default value.
[ WARN ] - [ org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:634) ] Using default maxIOWorkers
Hello ERROR ! ------ Test 500
Hello ERROR ! ------ Test 501
Hello ERROR ! ------ Test 502
Hello ERROR ! ------ Test 503
Hello ERROR ! ------ Test 504

8.5.查看Kafka的Topic

--进入node3，查看kafka的topic
cd /home/kafka-2.10/bin
./kafka-topics.sh --zookeeper node1,node2,node3 --list

可以看到，由于客户端代码的执行，Kafka里面的topic：flume-to-kafka被自动创建

8.6.启动Kafka Consumer:flume-to-kafka

我们在这里是查看topic： flume-to-kafka的消费信息

--进入node3，启动kafka消费者
cd /home/kafka-2.10/bin
./kafka-console-consumer.sh --zookeeper node1,node2,node3 --from-beginning --topic flume-to-kafka

控制台输出：

Hello ERROR ! ------ Test 500
Hello ERROR ! ------ Test 501
Hello ERROR ! ------ Test 502
Hello ERROR ! ------ Test 503
Hello ERROR ! ------ Test 504

8.7.创建Topic:storm-to-kafka

在Kafka里面创建另一个topic：

--进入node1，创建一个topic：storm-to-kafka
--设置3个partitions
--replication-factor=3
./kafka-topics.sh --zookeeper node1,node2,node3 --create --topic storm-to-kafka --partitions 3 --replication-factor 3

8.8.运行Storm代码

package storm.logfilter;

import java.util.ArrayList;
import java.util.List;
import java.util.Properties;

import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
import storm.kafka.bolt.KafkaBolt;
import storm.kafka.bolt.mapper.FieldNameBasedTupleToKafkaMapper;
import storm.kafka.bolt.selector.DefaultTopicSelector;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;

public class LogFilterTopology {

    public static class FilterBolt extends BaseBasicBolt {
        private static final long serialVersionUID = 1L;

        @Override
        public void execute(Tuple tuple, BasicOutputCollector collector) {
            String line = tuple.getString(0);
            // 包含ERROR的行留下
            if (line.contains("ERROR")) {
                System.err.println("Filter:  " + line + " ~ filtered.");
                collector.emit(new Values(line + " ~ filtered."));
            }
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            // 定义message提供给后面FieldNameBasedTupleToKafkaMapper使用
            declarer.declare(new Fields("message"));
        }
    }

    public static void main(String[] args) throws Exception {
        TopologyBuilder builder = new TopologyBuilder();

        // https://github.com/apache/storm/tree/master/external/storm-kafka
        // config kafka spout，话题
        String topic = "flume-to-kafka";
        ZkHosts zkHosts = new ZkHosts("node1:2181,node2:2181,node3:2181");
        // /MyKafka，偏移量offset的根目录，记录队列取到了哪里
        SpoutConfig spoutConfig = new SpoutConfig(zkHosts, topic, "/MyKafka", "MyTrack");
        List<String> zkServers = new ArrayList<String>();
        for (String host : zkHosts.brokerZkStr.split(",")) {
            zkServers.add(host.split(":")[0]);
        }

        spoutConfig.zkServers = zkServers;
        spoutConfig.zkPort = 2181;
        // 是否从头开始消费
        spoutConfig.forceFromStart = true;
        spoutConfig.socketTimeoutMs = 60 * 1000;
        // StringScheme将字节流转解码成某种编码的字符串
        spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());

        KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

        // set kafka spout
        builder.setSpout("kafkaSpout", kafkaSpout, 3);

        // set bolt
        builder.setBolt("filterBolt", new FilterBolt(), 8).shuffleGrouping("kafkaSpout");

        // 数据写出
        // set kafka bolt
        // withTopicSelector使用缺省的选择器指定写入的topic： storm-to-kafka
        // withTupleToKafkaMapper tuple==>kafka的key和message
        KafkaBolt kafka_bolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("storm-to-kafka"))
                .withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper());

        builder.setBolt("kafkaBolt", kafka_bolt, 2).shuffleGrouping("filterBolt");

        Config conf = new Config();
        // set producer properties.
        Properties props = new Properties();
        props.put("metadata.broker.list", "node1:9092,node2:9092,node3:9092");
        /**
         * Kafka生产者ACK机制 0 ： 生产者不等待Kafka broker完成确认，继续发送下一条数据 1 ：
         * 生产者等待消息在leader接收成功确认之后，继续发送下一条数据 -1 ：
         * 生产者等待消息在follower副本接收到数据确认之后，继续发送下一条数据
         */
        props.put("request.required.acks", "1");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        conf.put("kafka.broker.properties", props);

        conf.put(Config.STORM_ZOOKEEPER_SERVERS, zkServers);
        
        if (args == null || args.length == 0) {
            // 本地方式运行
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("storm-kafka-topology", conf, builder.createTopology());
        } else {
            // 集群方式运行
            conf.setNumWorkers(3);
            StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
        }
        
    }
}

8.9.启动Kafka Consumer:storm-to-kafka

我们在这里是查看topic： storm-to-kafka的消费信息

--进入node1，启动kafka消费者
cd /home/kafka-2.10/bin
./kafka-console-consumer.sh --zookeeper node1,node2,node3 --from-beginning --topic storm-to-kafka

控制台输出：

Hello ERROR ! ------ Test 504 ~ filtered.
Hello ERROR ! ------ Test 500 ~ filtered.
Hello ERROR ! ------ Test 501 ~ filtered.
Hello ERROR ! ------ Test 503 ~ filtered.
Hello ERROR ! ------ Test 502 ~ filtered.

========================================================