1.kafka日志结构
直接举例子:
例如kafka有个名字叫 haha 的topic,那么kafka日志下面有kafka-0,kafka-1,kafka-2...,kafka-n,具体多少个,创建分区的时候有几个分区,n就是几。
[root@haha kafka]# ls haha- haha-0/ haha-1/ haha-2/
每个目录下面有三个日志文件,xxx.log是消息集文件, xxx.index 偏移量索引文件 ,xxx.timeindex 时间戳索引文件
[root@haha kafka]# ls haha-1 00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
创建主题:
[root@haha kafka]# kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic log-format
[root@haha kafka]# ls /data/appData/kafka/log-format- log-format-0/ log-format-1/ log-format-2/ [root@haha kafka]# ls /data/appData/kafka/log-format-0 00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex
修改字段大小及索引跨度配置
[root@haha kafka]# kafka-topics.sh --zookeeper localhost:2181 --alter --topic log-format --config segment.bytes=1024 --config index.interval.bytes=100 WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic "log-format".
查看分topic区状态
[root@haha kafka]# /data/app/zookeeper/bin/zkCli.sh -server localhost:2181 [zk: localhost:2181(CONNECTED) 0] ls /brokers/ids [zk: localhost:2181(CONNECTED) 3] get /brokers/topics/log-format/partitions/1/state cZxid = 0x245 ctime = Thu Nov 01 11:29:47 CST 2018 mZxid = 0x245 mtime = Thu Nov 01 11:29:47 CST 2018 pZxid = 0x245 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 72 numChildren = 0
如果集群的日志则分布如下
通过kafka自带的脚本查看topic信息
[root@haha kafka]# kafka-topics.sh --describe --zookeeper localhost:2181 --topic log-format Topic:log-format PartitionCount:3 ReplicationFactor:1 Configs:index.interval.bytes=100,segment.bytes=1024 Topic: log-format Partition: 0 Leader: 5 Replicas: 5 Isr: 5 Topic: log-format Partition: 1 Leader: 5 Replicas: 5 Isr: 5 Topic: log-format Partition: 2 Leader: 5 Replicas: 5 Isr: 5
执行一下命令将二进制分段日志文件转化为字符类型的文件
[root@haha kafka]# kafka-run-class.sh kafka.tools.DumpLogSegments --files log-format-0/00000000000000000000.log --print-data-log Dumping log-format-0/00000000000000000000.log Starting offset: 0