• kafka 日志结构


    1.kafka日志结构

    直接举例子:

    例如kafka有个名字叫 haha 的topic,那么kafka日志下面有kafka-0,kafka-1,kafka-2...,kafka-n,具体多少个,创建分区的时候有几个分区,n就是几。

    [root@haha kafka]# ls haha-
    haha-0/ haha-1/ haha-2/

    每个目录下面有三个日志文件,xxx.log是消息集文件, xxx.index 偏移量索引文件 ,xxx.timeindex 时间戳索引文件

    [root@haha kafka]# ls haha-1
    00000000000000000000.index  00000000000000000000.log  00000000000000000000.timeindex
    

    创建主题:

    [root@haha kafka]# kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic log-format
    
    [root@haha kafka]# ls /data/appData/kafka/log-format-
    log-format-0/ log-format-1/ log-format-2/
    
    [root@haha kafka]# ls /data/appData/kafka/log-format-0
    00000000000000000000.index  00000000000000000000.log  00000000000000000000.timeindex
    

    修改字段大小及索引跨度配置

    [root@haha kafka]# kafka-topics.sh --zookeeper localhost:2181 --alter --topic log-format --config segment.bytes=1024 --config index.interval.bytes=100
    WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
             Going forward, please use kafka-configs.sh for this functionality
    Updated config for topic "log-format".
    

    查看分topic区状态

    [root@haha kafka]# /data/app/zookeeper/bin/zkCli.sh -server localhost:2181 
    
    [zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
    
    [zk: localhost:2181(CONNECTED) 3] get /brokers/topics/log-format/partitions/1/state
    cZxid = 0x245
    ctime = Thu Nov 01 11:29:47 CST 2018
    mZxid = 0x245
    mtime = Thu Nov 01 11:29:47 CST 2018
    pZxid = 0x245
    cversion = 0
    dataVersion = 0
    aclVersion = 0
    ephemeralOwner = 0x0
    dataLength = 72
    numChildren = 0
    

    如果集群的日志则分布如下

    通过kafka自带的脚本查看topic信息

    [root@haha kafka]# kafka-topics.sh --describe --zookeeper localhost:2181 --topic log-format
    Topic:log-format	PartitionCount:3	ReplicationFactor:1	Configs:index.interval.bytes=100,segment.bytes=1024
    	Topic: log-format	Partition: 0	Leader: 5	Replicas: 5	Isr: 5
    	Topic: log-format	Partition: 1	Leader: 5	Replicas: 5	Isr: 5
    	Topic: log-format	Partition: 2	Leader: 5	Replicas: 5	Isr: 5
    

    执行一下命令将二进制分段日志文件转化为字符类型的文件

    [root@haha kafka]# kafka-run-class.sh kafka.tools.DumpLogSegments --files log-format-0/00000000000000000000.log --print-data-log
    Dumping log-format-0/00000000000000000000.log
    Starting offset: 0
    
  • 相关阅读:
    Tabular DataStream protocol 协议
    Redis 分片实现 Redis Shard [www]
    进程线程协程那些事儿
    Linux下用freetds执行SQL Server的sql语句和存储过程
    unixODBC
    在linux下有没有什么软件可以连接windows上的MSSQL SERVER
    Nginx使用ssl模块配置HTTPS支持
    谈一款MOBA类游戏《码神联盟》的服务端架构设计与实现
    core dump使用方法、设置、测试用例
    linux下生成core dump文件方法及设置
  • 原文地址:https://www.cnblogs.com/sunshine-long/p/9886043.html
Copyright © 2020-2023  润新知