https://www.cnblogs.com/huxi2b/p/8316289.html
五、副本因子
__consumer_offest不受server.properties中num.partitions和default.replication.factor参数的制约。相反地，它的分区数和备份因子分别由offsets.topic.num.partitions和offsets.topic.replication.factor参数决定。这两个参数的默认值分别是50和1，表示该topic有50个分区，副本因子是1。鉴于位移和group元数据等信息都保存在该topic中，实际使用过程中很多用户都会将offsets.topic.replication.factor设置成大于1的数以增加可靠性，这是推荐的做法。不过在0.11.0.0之前，这个设置是有缺陷的：假设你设置了offsets.topic.replication.factor = 3，只要Kafka创建该topic时可用broker数<3，那么创建出来的__consumer_offsets的备份因子就是2。也就是说Kafka并没有尊重我们设置的offsets.topic.replication.factor参数。好在这个问题在0.11.0.0版本得到了解决，现在用户在使用时，一旦需要创建__consumer_offsets了Kafka一定会保证凑齐足量的broker才会开始创建，否则就抛出异常给用户。
以集中化环境为例，集中化kafka集群部署的机器为10.1.130.101，10.1.130.102，10.1.130.103，对应的broker.id(每台机器的server.properties中配置)分别为101，102，103
1、首先检查kafka配置文件config/server.properties
offsets.topic.replication.factor=3
该配置如果配置的是1，需要改为大于1的数
2、对于已经创建的__consumer_offsets需手动增加副本数，方法如下（都在kafka的bin目录下进行）：
第一步：创建规则json（注意这里的[101,102,103]里面对应的是三个broker.id，实际环境请对应修改）
cat > increase-replication-factor.json <<EOF
{"version":1, "partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":1,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":2,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":3,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":4,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":5,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":6,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":7,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":8,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":9,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":10,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":11,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":12,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":13,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":14,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":15,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":16,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":17,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":18,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":19,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":20,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":21,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":22,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":23,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":24,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":25,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":26,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":27,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":28,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":29,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":30,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":31,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":32,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":33,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":34,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":35,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":36,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":37,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":38,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":39,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":40,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":41,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":42,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":43,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":44,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":45,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":46,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":47,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":48,"replicas":[101,102,103]},
{"topic":"__consumer_offsets","partition":49,"replicas":[101,102,103]}]
}
EOF
第二步：执行，保证在所有节点启动的情况下进行
./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
第三步：验证
./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
最后查看__consumer_offsets的相关信息是否类似下面：
（注意：第一要看ReplicationFactor是否为3；第二Leader不能出现有-1的情况，即使是在一个节点挂掉后）
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 1 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 2 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 3 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 4 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 5 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 6 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 7 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 8 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 9 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 10 Leader: 103 Replicas: 101,102,103 Isr: 103,101,102
Topic: __consumer_offsets Partition: 11 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 12 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 13 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 14 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 15 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 16 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 17 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 18 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 19 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 20 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 21 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 22 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 23 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 24 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 25 Leader: 103 Replicas: 101,102,103 Isr: 103,101,102
Topic: __consumer_offsets Partition: 26 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 27 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 28 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 29 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 30 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 31 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 32 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 33 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 34 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 35 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 36 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 37 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 38 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 39 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 40 Leader: 103 Replicas: 101,102,103 Isr: 103,101,102
Topic: __consumer_offsets Partition: 41 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 42 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 43 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
Topic: __consumer_offsets Partition: 44 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 45 Leader: 102 Replicas: 101,102,103 Isr: 102,101,103
Topic: __consumer_offsets Partition: 46 Leader: 103 Replicas: 101,102,103 Isr: 103,101,102
Topic: __consumer_offsets Partition: 47 Leader: 101 Replicas: 101,102,103 Isr: 101,102,103
Topic: __consumer_offsets Partition: 48 Leader: 102 Replicas: 101,102,103 Isr: 102,103,101
Topic: __consumer_offsets Partition: 49 Leader: 103 Replicas: 101,102,103 Isr: 103,102,101
对于采集任务而言，可以故意去停掉一个节点的进程，然后修改采集配置，修改某个任务的周期或其他，看任务是否实时下发到对应组件上面