一、快速搭建Kafka环境
基于Docker容器创建（供参考）：
https://www.cnblogs.com/mindzone/p/15608984.html
这里简要写一下命令：
# 拉取zk + kafka的镜像 docker pull wurstmeister/zookeeper docker pull wurstmeister/kafka # 创建zk容器 docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper # 创建kafka容器 docker run -d --name kafka \ -p 9092:9092 \ -e KAFKA_BROKER_ID=0 \ -e KAFKA_ZOOKEEPER_CONNECT=Linux主机IP:2181 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://Linux主机IP:9092 \ -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \ -t wurstmeister/kafka # 检查kafka运行情况 docker ps
测试Topic消息是否正常生产和消费（注意终端是阻塞的，需要多开终端窗口测试）：
#窗口1 生产 [root@centos-linux ~]# docker exec -it kafka /bin/bash bash-4.4# kafka-console-producer.sh --broker-list localhost:9092 --topic topic名称 #窗口2 消费 [root@centos-linux ~]# docker exec -it kafka /bin/bash bash-4.4# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic名称 --from-beginning # 样例 bash-4.4# kafka-console-producer.sh --broker-list localhost:9092 --topic producer
bash-4.4# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic producer --from-beginning
二、配置Maxwell 绑定Kafka
1、方式一，简单命令参数启动：
cd /usr/local/maxwell-1.29.2 ./bin/maxwell \ --user='maxwell' \ --password='123456' \ --host='192.168.2.225' \ --port='3308' \ --producer=kafka \ --kafka.bootstrap.servers=localhost:9092 \ --kafka_topic=producer \ --jdbc_options='useSSL=false&serverTimezone=Asia/Shanghai'
Maxwell运行成功的输出：
[root@localhost maxwell-1.29.2]# ./bin/maxwell \ > --user='maxwell' \ > --password='123456' \ > --host='192.168.2.225' \ > --port='3308' \ > --producer=kafka \ > --kafka.bootstrap.servers=localhost:9092 \ > --kafka_topic=producer \ > --jdbc_options='useSSL=false&serverTimezone=Asia/Shanghai' Using kafka version: 1.0.0 14:13:50,533 INFO Maxwell - Starting Maxwell. maxMemory: 247332864 bufferMemoryUsage: 0.25 14:13:50,783 INFO ProducerConfig - ProducerConfig values: acks = 1 batch.size = 16384 bootstrap.servers = [localhost:9092] buffer.memory = 33554432 client.id = compression.type = snappy connections.max.idle.ms = 540000 enable.idempotence = false interceptor.classes = null key.serializer = class org.apache.kafka.common.serialization.StringSerializer linger.ms = 0 max.block.ms = 60000 max.in.flight.requests.per.connection = 5 max.request.size = 1048576 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner receive.buffer.bytes = 32768 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 0 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = null value.serializer = class org.apache.kafka.common.serialization.StringSerializer 14:13:50,847 INFO AppInfoParser - Kafka version : 1.0.0 14:13:50,847 INFO AppInfoParser - Kafka commitId : aaa7af6d4a11b29d 14:13:50,871 INFO Maxwell - Maxwell v1.29.2 is booting (MaxwellKafkaProducer), starting at Position[BinlogPosition[mysql-bin.000005:225424], lastHeartbeat=1642486284932] 14:13:51,040 INFO MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[mysql-bin.000005:16191], lastHeartbeat=0]) 14:13:51,205 INFO BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin.000005:225424 14:13:51,235 INFO BinaryLogClient - Connected to 192.168.2.225:3308 at mysql-bin.000005/225424 (sid:6379, cid:215) 14:13:51,235 INFO BinlogConnectorReplicator - Binlog connected.
2、方式二、写在配置文件中：
cd /usr/local/maxwell-1.29.2 vim config.properties
参数项：
kafka_topic=maxwell producer=kafka kafka.bootstrap.servers=localhost:9092 host=192.168.2.225 user=maxwell password=123456 port=3308
启动：
cd /usr/local/maxwell-1.29.2 ./bin/maxwell \ --config ./config.properties \ --jdbc_options='useSSL=false&serverTimezone=Asia/Shanghai'
三、Kafka监听测试
由Kafka监听后，maxwell不再打印信息，后台运行，交由kafka发送
在DB操作非查询SQL时，可以发现Kafka消费者能够收到消息
消费者终端的消息：
[root@localhost maxwell-1.29.2]# docker exec -it kafka /bin/bash bash-5.1# bash-4.4# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic producer --from-beginning bash: bash-4.4#: command not found bash-5.1# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic producer --from-beginning [2022-01-18 06:09:16,853] WARN [Consumer clientId=consumer-console-consumer-5789-1, groupId=console-consumer-5789] Error while fetching metadata with correlation id 2 : {producer=LEADER_NOT_AVAILABLE} (org.apache.kafka.cli [2022-01-18 06:09:16,987] WARN [Consumer clientId=consumer-console-consumer-5789-1, groupId=console-consumer-5789] Error while fetching metadata with correlation id 4 : {producer=LEADER_NOT_AVAILABLE} (org.apache.kafka.cli hello aaaaaaaaaaaaaaa {"database":"test-db","table":"day_sale","type":"delete","ts":1642486851,"xid":71876,"commit":true,"data":{"ID":166,"PRODUCT":"产品C","CHANNEL":"淘宝","AMOUNT":2497.0000,"SALE_DATE":"2022-01-18 13:48:48"}}
四、Kafka分区控制
1、用途：
希望kakfa能够并行执行，因为监听的消息都只送到一个分区的队列上，效率太慢
让Kafka进行并发发送，就多开分区进行，每个分区同时执行消息发送
2、问题：
教程并没有说明是如何关联库和分区的关系，只是会有不同
3、技术要点：
如何配置maxwell对kafka的分区？
参考config.properties对kafka配置的说明：
# *** kafka *** # list of kafka brokers #kafka.bootstrap.servers=hosta:9092,hostb:9092 # kafka topic to write to # this can be static, e.g. 'maxwell', or dynamic, e.g. namespace_%{database}_%{table} # in the latter case 'database' and 'table' will be replaced with the values for the row being processed #kafka_topic=maxwell # alternative kafka topic to write DDL (alter/create/drop) to. Defaults to kafka_topic #ddl_kafka_topic=maxwell_ddl -- 这段是关于分区的配置信息： # *** partitioning *** # 按照什么方式对数据进行划分？ # What part of the data do we partition by? # 参数项：库 表 主键 事务ID 线程ID 字段 # producer_partition_by=database # [database, table, primary_key, transaction_id, thread_id, column] # 如果选用字段来对数据进行划分， 指定在使用producer\u partition\u by=column时，分区依据的字段 # specify what fields to partition by when using producer_partition_by=column # column separated list. # 指明字段使用的是哪些 # producer_partition_columns=id,foo,bar # 如果指明的字段不存在，则会分区规则回退到库名进行划分 # when using producer_partition_by=column, partition by this when # the specified column(s) don't exist. # producer_partition_by_fallback=database # *** kinesis *** # kinesis_stream=maxwell # AWS places a 256 unicode character limit on the max key length of a record # http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html # # Setting this option to true enables hashing the key with the md5 algorithm # before we send it to kinesis so all the keys work within the key size limit. # Values: true, false # Default: false #kinesis_md5_keys=true
4、分区测试案例：
- 1、创建新的Topic并分配6个分区
# 进入kafka容器 docker exec -it kafka /bin/bash
# 创建主题并分配分区 （必须添加副本参数） kafka-topics.sh --zookeeper 192.168.177.129:2181 --topic maxwell --create --replication-factor 1 --partitions 6 副本数量 1 --replication-factor 1 分区数量 6 --partitions 6
- 2、更新maxwell配置（按字段配置很少，就按照库划分配置即可）
# kafka配置 producer=kafka kafka.bootstrap.servers=localhost:9092 # 改Topic名称 kafka_topic=maxwell # 改分区配置 producer_partition_by=database
- 3、重新启动maxwell
cd /usr/local/maxwell-1.29.2 ./bin/maxwell \ --config ./config.properties \ --jdbc_options='useSSL=false&serverTimezone=Asia/Shanghai'
- 4、向库中写入数据，然后查看kafka消息（使用Kafka tool可视化工具）
这一步省略具体步骤，只要是DML操作就行，效果查看使用【Kafka Tool】工具 （offset explorer）
五、关于Kafka分区配置的命令补充
Kafka基于这些命令脚本实现功能：
[root@localhost maxwell-1.29.2]# docker exec -it kafka ls /opt/kafka_2.13-2.8.1/bin connect-distributed.sh kafka-preferred-replica-election.sh connect-mirror-maker.sh kafka-producer-perf-test.sh connect-standalone.sh kafka-reassign-partitions.sh kafka-acls.sh kafka-replica-verification.sh kafka-broker-api-versions.sh kafka-run-class.sh kafka-cluster.sh kafka-server-start.sh kafka-configs.sh kafka-server-stop.sh kafka-console-consumer.sh kafka-storage.sh kafka-console-producer.sh kafka-streams-application-reset.sh kafka-consumer-groups.sh kafka-topics.sh kafka-consumer-perf-test.sh kafka-verifiable-consumer.sh kafka-delegation-tokens.sh kafka-verifiable-producer.sh kafka-delete-records.sh trogdor.sh kafka-dump-log.sh windows kafka-features.sh zookeeper-security-migration.sh kafka-leader-election.sh zookeeper-server-start.sh kafka-log-dirs.sh zookeeper-server-stop.sh kafka-metadata-shell.sh zookeeper-shell.sh kafka-mirror-maker.sh
语句执行报错
kafka-topics.sh --zookeeper 192.168.177.129:2181 --topic maxwell --create --replication-factor 2 --partitions 3 [2022-01-18 08:19:44,532] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 4 larger than available brokers: 1.
报错思路分析
https://www.cnblogs.com/tyoutetu/p/10855283.html
# 即，需要Kafka集群， 一个Kafka代表一个broker，副本必须小于等于集群的数量
--replication-factor （指定数量必须小于等于Kafka集群数，如果单个，写1即可）
不能修改分区数量的原因：
# 分区的数量只能增加，不能减少 bash-5.1# kafka-topics.sh --zookeeper 192.168.177.129:2181 -alter --partitions 3 --topic maxwell WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected Error while executing topic command : The number of partitions for a topic can only be increased. Topic maxwell currently has 6 partitions, 3 would not be [2022-01-18 08:28:42,743] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The number of partitions for a topic can only be increased. Topi (kafka.admin.TopicCommand$)
解决办法：
删除主题 -> 重建主题