• kafka consumer 自动提交 offset


    org.apache.kafka.clients.consumer.KafkaConsumer#pollOnce

    private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollOnce(long timeout) {
        client.maybeTriggerWakeup();
    
        long startMs = time.milliseconds();
        // 这里面触发自动提交
        coordinator.poll(startMs, timeout);
    
        // Lookup positions of assigned partitions
        boolean hasAllFetchPositions = updateFetchPositions();
    
        // 对拉取到的数据,更新 position
        Map<TopicPartition, List<ConsumerRecord<K, V>>> records = fetcher.fetchedRecords();
        if (!records.isEmpty())
            return records;
    
        // 发送拉取数据请求
        fetcher.sendFetches();
    
        long nowMs = time.milliseconds();
        long remainingTimeMs = Math.max(0, timeout - (nowMs - startMs));
        long pollTimeout = Math.min(coordinator.timeToNextPoll(nowMs), remainingTimeMs);
    
        // We do not want to be stuck blocking in poll if we are missing some positions
        // since the offset lookup may be backing off after a failure
        if (!hasAllFetchPositions && pollTimeout > retryBackoffMs)
            pollTimeout = retryBackoffMs;
    
        client.poll(pollTimeout, nowMs, new PollCondition() {
            @Override
            public boolean shouldBlock() {
                // since a fetch might be completed by the background thread, we need this poll condition
                // to ensure that we do not block unnecessarily in poll()
                return !fetcher.hasCompletedFetches();
            }
        });
    
        // after the long poll, we should check whether the group needs to rebalance
        // prior to returning data so that the group can stabilize faster
        if (coordinator.needRejoin())
            return Collections.emptyMap();
    
        return fetcher.fetchedRecords();
    }

    结论就是:consumer 拉取到消息后,会更新保存的位点信息,下次拉取消息前,若自动提交的时间到了,就会把位点信息提交到 broker。

  • 相关阅读:
    Android源码分析(二)-----如何编译修改后的framework资源文件
    Android源码分析(一)-----如何快速掌握Android编译文件
    AI2(App Inventor 2)离线版服务器网络版
    AI2(App Inventor 2)离线版服务器单机版
    AI2(App Inventor 2)离线版服务器(2019.04.28更新)
    解释器模式
    迭代器模式
    备忘录模式
    访问者模式
    命令模式
  • 原文地址:https://www.cnblogs.com/allenwas3/p/11473827.html
Copyright © 2020-2023  润新知