• 大叔问题定位分享(47)kafka执行reassign后部分partition一直is still in progress


    背景

    kafka执行reassign后部分partition一直is still in progress,日志如下:

    定位

    经排查发现问题原因是做reassign的时候,将分区从相同broker的一个盘移动到另一个盘,这个操作目前还不被支持,官方解释如下

    1. How to move replica between log directories on the same broker
      Problem statement:
      Kafka doesn’t not allow user to move replica to another log directory on the same broker in runtime. This is not needed previously because we use RAID-10 and the load is already balanced across disks. But it will be needed to use JBOD with Kafka.
      Currently a replica only has two possible states, follower or leader. And it is identified by the 3-tuple (topic, partition, broker). This works for the current implementation because there can be at most one such replica on a broker. However, now we can have two such replicas on a broker when we move replica from one log directory to another log directory on the same broker. Either a replica should be identified by its log directory as well, or the broker needs to persist information under the log directory to tell the destination replica from source replica that is being moved.
      In addition, user needs to be able to query list of partitions and their size per log directory on any machine so that they can determine how to move replicas to balance the load. While these information may be retrieved via external tools if user can ssh to the machine that is running the Kafka broker, development of such tools may not be easy on all the operating systems that Kafka may run on. Further, a user may not even have the authorization to access the machine. Therefore Kafka needs to provide new request/response to provide this information to users.
      Solution:
      The idea is that user can send a AlterReplicaDirRequest which tells broker to move topicPartition directory (which contains all log segments of the topicPartition replica) from the source log directory to a destination log directory. Broker can create a new directory with .move postfix on the destination log directory to hold all log segments of the replica. This allows broker to tell log segments of the replica on the destination log directory from log segments of the replica on the source log directory during broker startup. The broker can create new log segments for the replica on the destination log directory, push data from source log to the destination log, and replace source log with the destination log for this replica once the new log has caught up.

    大意为,一个replica有2种状态,follower或者leader,而一个replica是通过topic-partition-broker来区分,注意这里没有到数据盘或者路径,如果在一个broker上从一个盘迁移到另一个盘,这时replica的状态就会出现冲突。
    这时的解决方法是使用AlterReplicaDirRequest。

    官方文档

  • 相关阅读:
    selenium2基本控件介绍及其代码
    selenium2元素定位Xpath和cssSelector
    Selenium2启动浏览器且加载插件
    bash之条件测试if/else
    bash脚本编写基础
    Android后台的linux一直保持唤醒状态,不进入睡眠
    Linux任务计划、周期性任务执行
    C#进阶系列——WebApi 跨域问题解决方案:CORS(转载)
    linq group by
    echart 分组属性
  • 原文地址:https://www.cnblogs.com/barneywill/p/16289437.html
Copyright © 2020-2023  润新知