前面我们分别介绍了kafka的相关基本原理,kafka的集群服务器搭建以及kafka相关的配置,本文综合前面的理论知识,运用kafka Java API实现一个简单的客户端Demo。
开发环境
- 操作系统:MacOS 10.12.3
- 开发平台:Eclipse Neon.2 Release (4.6.2)
- JDK: java version 1.8.0_121
- zookeeper: zookeeper-3.4.9
- kafka: kafka-2.10-0.10.2.0
项目的建立与实现
首先为大家展示一下项目最终的结构图,如下:
下面开始建立项目:
- 首先建立一个基本的Maven Java Project 项目框架,项目名称为 kafkaDemo,建立项目流程参考:maven 基本框架搭建;
- 然后修改pom.xml文件内容,为项目引入kafka 客户端jar包:
<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.10.2.0</version> </dependency>
添加完成后保存pom.xml,然后maven update project。当update完成后,maven依赖包里的jar包应该如上图所示。
下面分别添加producer和consumer客户端代码。
在src/main/java目录下新建package,命名为 com.unionpay.producer。由于kafka producer端有同步发送和异步发送之分,本项目将两个示例都进行展示,首先编写同步发送ProducerSync代码。
ProducerSync.java:
1 package com.unionpay.producer; 2 3 import java.util.Properties; 4 5 import org.apache.kafka.clients.producer.KafkaProducer; 6 import org.apache.kafka.clients.producer.Producer; 7 import org.apache.kafka.clients.producer.ProducerRecord; 8 9 10 public class ProducerSync { 11 12 private static final String TOPIC = "my-replicated-topic"; 13 public static void main(String[] args) { 14 // TODO Auto-generated method stub 15 16 Properties properties = new Properties(); 17 //客户端用于建立与kafka集群连接的host:port组,如果有多个broker,则用“,”隔开 18 // "host1:port1,host2:port2,host3,post3" 19 properties.put("bootstrap.servers", "127.0.0.1:9092"); 20 21 // producer在向servers发送信息后,是否需要serveres向客户端(producer)反馈接受消息状态用此参数配置 22 // acks=0:表示producer不需要等待集群服务器发送的确认消息;acks=1:表示producer需要等到topic对应的leader发送的消息确认; 23 // acks=all:表示producer需要等到leader以及所有followers的消息确认,这是最安全的消息保障机制 24 properties.put("acks", "all"); 25 properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 26 properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 27 properties.put("buffer.memory", "33554432"); 28 29 Producer<String,String> producer = new KafkaProducer<String,String>(properties); 30 31 for(int i=0;i<100;i++){ 32 33 String message = "Sync : this is the " + i + "th message for test!"; 34 ProducerRecord<String, String> producerRecord = new ProducerRecord<String, String>(TOPIC, message); 35 producer.send(producerRecord); 36 37 try { 38 Thread.sleep(1000); 39 } catch (InterruptedException e) { 40 // TODO Auto-generated catch block 41 e.printStackTrace(); 42 } 43 } 44 45 producer.close(); 46 47 } 48 49 }
然后编写异步ProducerAsync代码。
ProducerAsync.java:
1 package com.unionpay.producer; 2 3 import java.util.Properties; 4 5 import org.apache.kafka.clients.producer.KafkaProducer; 6 import org.apache.kafka.clients.producer.Producer; 7 import org.apache.kafka.clients.producer.ProducerRecord; 8 9 public class ProducerAsync { 10 11 private static final String TOPIC = "my-replicated-topic"; 12 public static void main(String[] args) { 13 // TODO Auto-generated method stub 14 15 Properties props = new Properties(); 16 props.put("bootstrap.servers", "127.0.0.1:9092"); 17 props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 18 props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 19 props.put("producer.type", "async"); 20 props.put("batch.size", "16384"); 21 22 Producer<String,String> producer = new KafkaProducer<String,String>(props); 23 24 for(int i=0;i<100;i++){ 25 26 String message = "Async : this is the " + i + "th message for test!"; 27 28 ProducerRecord producerRecord = new ProducerRecord(TOPIC, message); 29 producer.send(producerRecord); 30 31 try { 32 Thread.sleep(1000); 33 } catch (InterruptedException e) { 34 // TODO Auto-generated catch block 35 e.printStackTrace(); 36 } 37 } 38 39 producer.close(); 40 } 41 }
从两个代码文件比对来看,异步中多了一句配置语句props.put("producer.type", "async");
然后编写consumer端代码
GroupConsumer.java:
1 package com.unionpay.consumer; 2 3 import java.util.Arrays; 4 import java.util.Properties; 5 6 import org.apache.kafka.clients.consumer.ConsumerRecord; 7 import org.apache.kafka.clients.consumer.ConsumerRecords; 8 import org.apache.kafka.clients.consumer.KafkaConsumer; 9 10 public class GroupConsumer { 11 12 private static final String BROKER = "127.0.0.1:9092"; 13 private static final String TOPIC = "my-replicated-topic"; 14 15 16 public static void main(String[] args) { 17 // TODO Auto-generated method stub 18 19 Properties props = new Properties(); 20 props.put("bootstrap.servers",BROKER); 21 // 用来唯一标识consumer进程所在组的字符串,如果设置同样的group id,表示这些processes都是属于同一个consumer group 22 props.put("group.id", "group1"); 23 // 如果为真,consumer所fetch的消息的offset将会自动的同步到zookeeper。这项提交的offset将在进程挂掉时,由新的consumer使用 24 props.put("enable.auto.commit", "true"); 25 // consumer向zookeeper提交offset的频率 26 props.put("auto.commit.interval.ms", "1000"); 27 props.put("session.timeout.ms", "30000"); 28 props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); 29 props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); 30 31 KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props); 32 33 // 订阅topic,可以为多个用,隔开Arrays.asList("topic1","topic2"); 34 consumer.subscribe(Arrays.asList(TOPIC)); 35 36 while(true){ 37 ConsumerRecords<String,String> consumerRecords = consumer.poll(100); 38 39 for(ConsumerRecord<String,String> consumerRecord : consumerRecords){ 40 System.out.println(consumerRecord.value()); 41 } 42 } 43 } 44 }
到目前为止,我们的项目建立完成啦,下面启动zookeeper集群服务器,启动kafka集群服务器:
//启动zookeeper集群服务器 cd ~/DevelopEnvironment/zookeeper-3.4.9-kafka/bin ./zkServer.sh start //启动kafka集群服务器 cd ~/DevelopEnvironment/kafka_2.10-0.10.2.0/bin ./kafka-server-start.sh ../config/server.properties ./kafka-server-start.sh ../config/server-1.properties ./kafka-server-start.sh ../config/server-2.properties
当zookeeper集群服务器和kafka集群服务器启动成功后,然后分别运行GroupConsumer.java和ProducerAsync.java,客户端获取如下信息:
然后运行ProducerSync.java,客户端获取如下信息:
到此,游戏结束,我们的kafka API 使用demo介绍到此结束。