一、Kafka Broker 工作流程
1.1 ZooKeeper中存储的kafka信息
补充说明下/kafka/controller, 每个broker模块都将有一个controller模块,在竞争leader时谁先抢到/kafka/controller节点,谁来节点谁是leader。因此说/kafka/controller节点只是辅助作用。
1.2 Kafka Broker总体工作流程
二、Kafka Broker 节点服役和退役
2.1 服役新节点
新节点准备
执行均衡分区(数据转移)操作
1)创建需要数据转移的topic
vim topics-to-move.json { "topics": [ {"topic": "first"} ], "version": 1 }
2) 生成一个负载均衡的计划 :
- bin/kafka-reassign-partitions.sh 执行均衡分区的脚本;
- --topics-to-move-json-file topics-to-move.json 需要均衡的topic是上一步创建的
- -broker-list "0,1,2,3" 参与均衡分区的broker有
- --generate代表生成计划,并未执行
bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
返回结果
//未修改前的分区负载情况 Current partition replica assignment {"version":1,"partitions":[{"topic":"first","partition":0,"replic as":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","par tition":1,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"to pic":"first","partition":2,"replicas":[1,0,2],"log_dirs":["any"," any","any"]}]} //计划修改为如下分区负载情况 Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"first","partition":0,"replic as":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","par tition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"to pic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any"," any","any"]}]}
3) 创建副本存储计划(把上一步 “计划修改为如下分区负载情况” 拷贝下来)
vim increase-replication-factor.json
{ "version":1, "partitions":[ {"topic":"first", "partition":0, "replicas":[2,3,0], "log_dirs":["any","any","any"] }, {"topic":"first", "partition":1, "replicas":[3,0,1], "log_dirs":["any","any","any"] }, {"topic":"first", "partition":2, "replicas":[0,1,2], "log_dirs":["any","any","any"] }] }
4) 执行副本存储计划:
bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
5)验证副本存储计划
bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
返回结果
Status of partition reassignment: Reassignment of partition first-0 is complete. Reassignment of partition first-1 is complete. Reassignment of partition first-2 is complete. Clearing broker-level throttles on brokers 0,1,2,3 Clearing topic-level throttles on topic first
2.2 退役旧节点
和2.1节类似,只是创建执行计划时,改为 --broker-list "0,1,2" 即可。
具体参考:https://achang.blog.csdn.net/article/details/123054073