Apache Kafka（四）- 使用 Java 访问 Kafka

1. Produer

1.1. 基本 Producer

首先使用 maven 构建相关依赖，这里我们服务器kafka 版本为 2.12-2.3.0，pom.xml 文件为：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.github.tang</groupId>
    <artifactId>kafka-beginner</artifactId>
    <version>1.0</version>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.3.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.26</version>
        </dependency>

    </dependencies>

</project>

然后创建一个 Producer：

package com.github.tang.kafka.tutorial1;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class ProducerDemo {

    private static String bootstrapServers = "server_xxx:9092";

    public static void main(String[] args) {

        /**
         * create Producer properties
         *
         * Properties are available in official document:
         * https://kafka.apache.org/documentation/#producerconfigs
         *
         */
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        // create the producer
        KafkaProducer<String, String> produer = new KafkaProducer<String, String>(properties);


        // create a producer record
        ProducerRecord<String, String> record =
                new ProducerRecord<String, String>("first_topic", "message from java");

        // send data - asynchronous
        /**
         *  asynchronous means the data would not send immediately
         *  however, the program would terminate immediately after run the send() method
         *  hence the data would not send to kafka topic
         *  and the consumer would not receive the data
         *
         *  so we need flush()
         */
        produer.send(record);

        /**
         *  use flush() to wait sending complete
         */
        produer.flush();
        produer.close();

    }
}

运行此程序可以在consumer-console-cli 下看到发送的消息。

1.2. 带Callback() 的Producer

Callback() 函数会在每次发送record 后执行，例如：

首先实例化一个 logger() 对象：

// create a logger
final Logger logger = LoggerFactory.getLogger(ProducerDemoCallback.class);

使用 Callback()：

/**
 * send data with Callback()
 */
for(int i = 0; i < 10; i++) {
    // create a producer record
    ProducerRecord<String, String> record =
            new ProducerRecord<String, String>("first_topic", "message from java" + Integer.toString(i));

    produer.send(record, new Callback() {
        public void onCompletion(RecordMetadata recordMetadata, Exception e) {
            // execute every time a record is successfully sent or an exception is thrown
            if (e == null) {
                // the record is sent successfully
                logger.info("Received new metadata. 
" +
                        "Topic: " + recordMetadata.topic() + "
" +
                        "Partition: " + recordMetadata.partition() + "
" +
                        "Offset: " + recordMetadata.offset() + "
" +
                        "Timestamp: " + recordMetadata.timestamp());
            } else {
                logger.error("Error while producing", e);
            }
        }
    });
}

部分输出结果如下：

[kafka-producer-network-thread | producer-1] INFO com.github.tang.kafka.tutorial1.ProducerDemoCallback - Received new metadata.

Topic: first_topic

Partition: 2

Offset: 21

Timestamp: 1565501879059

[kafka-producer-network-thread | producer-1] INFO com.github.tang.kafka.tutorial1.ProducerDemoCallback - Received new metadata.

Topic: first_topic

Partition: 2

Offset: 22

Timestamp: 1565501879075

1.3. 发送带key的records

上面的例子均是未带key，所以消息是按轮询的方式发送到partition。下面是带key的producer例子，重载send() 方法即可：

String key = "id_" + Integer.toString(i);

ProducerRecord<String, String> record =
        new ProducerRecord<String, String>(topic, key,"message from java" + Integer.toString(i));

2. Consumer

2.1. 基本Consumer

下面是一个基本的consumer 例子：

package com.github.tang.kafka.tutorial1;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;

public class ConsumerDemo {
    private static String bootstrapServers = "server:9092";
    private static String groupId = "my-forth-app";
    private static String topic = "first_topic";


    public static void main(String[] args) {
        Logger logger = LoggerFactory.getLogger(ConsumerDemo.class);

        /**
         * create Consumer properties
         *
         * Properties are available in official document:
         * https://kafka.apache.org/documentation/#consumerconfigs
         *
         */
        Properties properties = new Properties();
        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // create consumer
        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties);

        // subscribe consumer to our topic(s)
        consumer.subscribe(Arrays.asList(topic));

        // poll for new data
        while(true){
            ConsumerRecords<String, String> records =
                    consumer.poll(Duration.ofMinutes(100));

            for(ConsumerRecord record : records){
                logger.info("Key: " + record.key() + "	" + "Value: " + record.value() +
                        "Topic: " + record.partition() + "	" + "Partition: " + record.partition()
               );

            }
        }

    }
}

部分输出结果如下：

 

从输出结果可以看到，consumer 在读取时，（在指定offset为earliest的情况下）是先读完一个partition后，再读下一个partition。

2.2. Consumer balancing

之前提到过，在一个consumer group中的consumers可以自动做负载均衡。下面我们启动一个consumer后，再启动一个consumer。

下面是第一个consumer的日志：

 

在第二个consumer加入后，第一个consumer 重新分配 partition，从之前负责三个partition（0，1，2）到现在负责一个partition（2）。

对于第二个consumer的日志：

 

可以看到第二个consumer在加入后，开始负责2个partition（0与1）的读

2.3 Consumer 多线程方式：

package com.github.tang.kafka.tutorial1;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class ConsumerDemoWithThreads {

    private static Logger logger = LoggerFactory.getLogger(ConsumerDemoWithThreads.class);

    public static void main(String[] args) {
        String bootstrapServers = "server:9092";
        String groupId = "my-fifth-app";
        String topic = "first_topic";

        // latch for dealing with multiple threads
        CountDownLatch latch = new CountDownLatch(1);

        ConsumerRunnable consumerRunnable = new ConsumerRunnable(latch,
                bootstrapServers,
                groupId,
                topic);

        Thread myConsumerThread = new Thread(consumerRunnable);
        myConsumerThread.start();

        // add a shutdown hook
        Runtime.getRuntime().addShutdownHook(new Thread(() ->{
            logger.info("Caught shutdown hook");
            consumerRunnable.shutdown();

            try{
                latch.await();
            } catch (InterruptedException e){
                e.printStackTrace();
            }
            logger.info("Application has exited");

        }));

        try{
            latch.await();
        } catch (InterruptedException e){
            logger.error("Application got interrupted", e);
        } finally {
            logger.info("Application is closing");
        }

    }

    private static class ConsumerRunnable implements Runnable{

        private CountDownLatch latch;
        KafkaConsumer<String, String> consumer;
        private String bootstrapServers;
        private String topic;
        private String groupId;

        public ConsumerRunnable(CountDownLatch latch,
                              String bootstrapServers,
                              String groupId,
                              String topic){
            this.latch = latch;
            this.bootstrapServers = bootstrapServers;
            this.topic = topic;
            this.groupId = groupId;
        }

        @Override
        public void run() {

            Properties properties = new Properties();
            properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
            properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
            properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
            properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId);
            properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

            consumer = new KafkaConsumer<String, String>(properties);
            consumer.subscribe(Arrays.asList(topic));

            // poll for new data
            try {
                while (true) {
                    ConsumerRecords<String, String> records =
                            consumer.poll(Duration.ofMinutes(100));

                    for (ConsumerRecord record : records) {
                        logger.info("Key: " + record.key() + "	" + "Value: " + record.value());
                        logger.info("Partition: " + record.partition() + "	" + "Offset: " + record.offset()
                        );

                    }
                }
            } catch (WakeupException e){
                logger.info("Received shutdown signal!");
            } finally {
                consumer.close();

                // tell our main code we're done with the consumer
                latch.countDown();
            }
        }

        public void shutdown(){
            // the wakeup() method is a special method to interrupt consumer.poll()
            // it will throw the exceptioin WakeUpException
            consumer.wakeup();
        }
    }
}

2.4. Consumer使用 Assign and Seek

Consumer 中可以使用Assign 分配一个topic的partition，然后用seek方法从给定offset读取records。一般此方式用于replay数据或是获取一条特定的record。

在实现时，基于上一个例子，修改run()方法部分代码如下：

// assign and seek are most used to replay data or fetch a specific message

// assign
TopicPartition partitionToReadFrom = new TopicPartition(topic, 0);
long offsetToReadFrom = 15L;
consumer.assign(Arrays.asList(partitionToReadFrom));

// seek
consumer.seek(partitionToReadFrom, offsetToReadFrom);

int numberOfMessagesToRead = 5;
boolean keepOnReading = true;
int numberOfMessagesReadSoFar = 0;

// poll for new data
try {
    while (keepOnReading) {
        ConsumerRecords<String, String> records =
                consumer.poll(Duration.ofMinutes(100));

        for (ConsumerRecord record : records) {
            numberOfMessagesReadSoFar += 1;

            logger.info("Key: " + record.key() + "	" + "Value: " + record.value());
            logger.info("Partition: " + record.partition() + "	" + "Offset: " + record.offset()
            );

            if (numberOfMessagesReadSoFar >= numberOfMessagesToRead){
                keepOnReading = false;
                break;
            }
        }
    }
} catch (WakeupException e){
    logger.info("Received shutdown signal!");
} finally {
    consumer.close();

    // tell our main code we're done with the consumer
    latch.countDown();
}

需要注意的是，使用此方法时，不需要指定consumer group。

3. 客户端双向兼容

在Kafka 0.10.2 版本之后，Kafka 客户端与Kafka brokers可以实现双向兼容（通过将API版本化实现，也就是说：不同的版本客户端发送的API版本不一样，且服务端可以处理不同版本API的请求）。

也就是说：

• 一个老版本的客户端（1.1之前版本）可以与更新版本的broker（2.0版本）正常交互
• 一个新版本的客户端（2.0之前版本）可以与一个老版本的broker（1.1版本）正常交互

对此的建议是：在任何时候都是用最新的客户端lib版本。