3 Kafka两个版本在Zookeeper的元数据存储

0.8

[consumers, admin, config, controller, brokers, controller_epoch]

这些节点都归于谁管理

• consumers: 0.8版本的客户端Consumer
• admin:管理命令的脚本:kafka-preferred-replica-election.sh,
• config:管理命令的脚本:kafka-topics.sh
• controller:所有的broker监控这个。
• brokers:kafka-topics.sh建立topic
• controller_epoch:

1.consumers

1.1 ids 代表消费者线程

/consumers/[groupId]/ids/[consumerIdString]:每个consumer都有一个唯一的ID(consumerId可以通过配置文件指定,也可以由系统生成),此id用来标记消费者信息.

[zk: 10.39.1.66:22181(CONNECTED) 17] ls /consumers/guest_tblog_expo/ids
[guest_tblog_expo_yz4834.hadoop.data.sina.com.cn-1472452954131-13232511, guest_tblog_expo_yz4879.hadoop.data.sina.com.cn-1472452955003-3ff3d939]

这两个节点是临时的znode。

1.2 Consumer owner 表示partition和consumer线程的对应关系

/consumers/[groupId]/owners/[topic]/[partitionId] -> consumerIdString + threadId 索引编号

[zk: 10.39.1.66:22181(CONNECTED) 23] get /consumers/guest_tblog_expo/owners/openapi_exposure/0
guest_tblog_expo_yz4845.hadoop.data.sina.com.cn-1472452954464-2f40ad23-0

1.3 Consumer offset 表示partition被消费到的Offset位置

用来跟踪每个consumer目前所消费的partition中最大的offset
此znode为持久节点,可以看出offset跟group_id有关,以表明当消费者组(consumer group)中一个消费者失效,
重新触发balance,其他consumer可以继续消费.

/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)

[zk: 10.39.1.66:22181(CONNECTED) 26] get  /consumers/guest_tblog_expo/offsets/openapi_exposure/0
50793695368

2 admin 表示Controller管理的命令

常有的目录结构

• /admin/reassign_partitions 重新分配partition
• /admin/preferred_replica_election 重新preffed partition leader
• /admin/delete_topics 删除topic

3 config 配置

主要包括三个的配置

• topic : 存储topic集群属性之外的属性，比如修改了一个新的topic属性
• clients：Client and Topic configuration overrides: The content of both znodes has the same structure（consumer或者Producer的配置地方）
• changes: Config Change notification (Topic and Client config修改通知的地方)

clients 和 changes 在0.8 版本中也许没有。

[zk: localhost:2181(CONNECTED) 3] get /kafka10/config/topics/test2
{"version":1,"config":{"retention.ms":"3600000"}}

4 controller

/controller -> int (broker id of the controller) 存储center controller中央控制器所在kafka broker的信息

• 0.8 版本的存储内容

[zk: 10.39.1.66:22181(CONNECTED) 34] get /controller
3

• 0.10 版本的存储内容

[zk: localhost:2181(CONNECTED) 12] get /kafka10/controller
{"version":1,"brokerid":40107,"timestamp":"1479456552014"}

两个版本的存储内容有差别了。

6 controller_epoch 表示controller选举的次数

/controller_epoch -> int (epoch) 此值为一个数字,kafka集群中第一个broker第一次启动时为1，以后只要集群中center controller中央控制器所在broker变更或挂掉，就会重新选举新的center controller，每次center controller变更controller_epoch值就会 + 1;

[zk: localhost:2181(CONNECTED) 13] get /kafka10/controller_epoch
9

• 什么时候会重新选举controller？ 有broker与zk失去联系的时候。

5 brokers

存集群broker和topic的元信息:

• /brokers/topics/[topic] :存储某个topic的partitions所有分配信息
• ``

5.1 topic注册信息

• /brokers/topics/[topic]: 存储某个topic的partitions所有分配信息
• /brokers/topics/[topic]/partitions/[0...N] : partition的个数
• /brokers/topics/[topic]/partitions/[partitionId]/state ： partition的信息

[zk: localhost:2181(CONNECTED) 20] get /kafka10/brokers/topics/test2/partitions/0/state
{"controller_epoch":9,"leader":40106,"version":1,"leader_epoch":13,"isr":[40106,40107]}

[zk: 10.39.1.66:22181(CONNECTED) 38] get /brokers/topics/openapi_advertise/partitions/0/state
{ "controller_epoch":33, "isr":[ 13, 7 ], "leader":13, "leader_epoch":29, "version":1 }

5.2 Broker注册信息

/brokers/ids/[0...N] : 每个broker的配置文件中都需要指定一个数字类型的id(全局不可重复),此节点为临时znode(EPHEMERAL)

0.10 和 0.8 信息不一样。

[zk: 10.39.1.66:22181(CONNECTED) 39] get /brokers/ids/48152
{ "host":"yz48152.hadoop.data.sina.com.cn", "jmx_port":9999, "port":19092, "version":1 }

[zk: localhost:2181(CONNECTED) 22] get /kafka10/brokers/ids/4099
{"jmx_port":9999,"timestamp":"1479456552292","endpoints":["PLAINTEXT://10.39.40.99:9110"],"host":"10.39.40.99","version":3,"port":9110}

5.3 seqs 0.10 版本特有

未知？？？？

0.10

[consumers, admin, config, controller, brokers, controller_epoch, isr_change_notification, kafka-acl, kafka-acl-changes]

10.1 isr_change_notification

0.10版本多了3组znode

• /isr_change_notification/isr_change_x: Gets created when ISR is changed at any broker, controller watches for these notifications and sends MetadataUpdateRequest to all brokers. (ISR改变的时候，通知Controller，发送更新请求给all brokers.)

10.2 kafka-acl

关于授权，一共有三种：

• Cluster:Create集群权限（官方写的是“Create on Cluster”）
• Group: ConsumerGroup权限
• Topic：读写权限

[zk: localhost:2181(CONNECTED) 10] get /kafka10/kafka-acl/Cluster/kafka-cluster
{"version":1,"acls":[{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"*"},,{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"10.77.*"}]}

[zk: localhost:2181(CONNECTED) 12] get /kafka10/kafka-acl/Group/superConsumer
{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"}]}

[zk: localhost:2181(CONNECTED) 13] get /kafka10/kafka-acl/Topic/client_ad_realread
{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:fengchao1","permissionType":"Allow","operation":"Describe","host":"*"}]}

10.3 kafka-acl-changes

存储acl的变更信息。由授权的脚本触发。

[zk: localhost:2181(CONNECTED) 17] get /kafka10/kafka-acl-changes/acl_changes_0000000053
Topic:test2

关于这些节点里的内容格式，请查看kafka zk wiki