kafka简单使用

kafka的API基本上上就是消费者和生产者，可以使用不同语言操作这些API，也有很多组件，框架可以作为kafka的生产者或消费者，下面简单介绍几种

1、java

生产者
public class KafkaProducer {

public static void main(String[] args) throws IOException, InterruptedException {
    String topic = "topic_test";
    Properties props = new Properties();
    props.setProperty("bootstrap.servers", "node1:9092,node2:9092,node3:9092");
    props.setProperty("key.serializer", StringSerializer.class.getName());
    props.setProperty("value.serializer", StringSerializer.class.getName());
    KafkaProducer<String, String> producer = new KafkaProducer<>(props);

    while(true) {
        String value = "";
        ......
        Thread.sleep(1000L);
        System.out.println(topic);
        producer.send(new ProducerRecord<>(topic, value));
        System.out.println(value);
    }
    producer.close();
}


消费者
public class KafkaProducer {

public static void main(String[] args) throws IOException, InterruptedException {
    String topic = "topic_test";
    Properties props = new Properties();
    props.put("bootstrap.servers", servers);
    props.put("group.id", groupId);
    props.put("enable.auto.commit", "true");
    props.put("auto.commit.interval.ms", "1000");
    props.put("session.timeout.ms", "30000");
    props.put("auto.offset.reset", "earliest");
    props.put("key.deserializer", StringDeserializer.class.getName());
    props.put("value.deserializer", StringDeserializer.class.getName());
    
    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
    String topic = "topic_test";
    consumer.subscribe(Arrays.asList(topic));
    
    
    try {
         while(true) {
        ConsumerRecords<String, String> msgList = consumer.poll(1000);
            if(null != msgList && msgList.count() > 0){
                for (ConsumerRecord<String, String> record : msgList) {
                    System.out.println("=======receive: key = " + record.key() + ", value = " + record.value()+" offset==="+record.offset());
                }
            } else {
                Thread.sleep(1000);
            }
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    } finally {
        consumer.close();
    }

}

python,scala等其他语言也可以，有对应的API

一些组件如：flume，可以消费kafka的数据也可以作为生产端，filebeat也可以，这是采集类的 。

计算组件的如，presto有connect操作kafka的数据，sparkStreaming流计算，消费kafka，flink有kafka的souce和sink，之后补充