向导地址
https://debezium.io/documentation/reference/1.2/tutorial.html
1.启动kafka connector
$ docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka --link mysql:mysql debezium/connect:1.2
按照文档说法,运行下面命令可见结果
$ curl -H "Accept:application/json" localhost:8083/ {"version":"2.5.0","commit":"cb8625948210849f"}
但发现绑定8083不成功,定位半天发现connector绑定在pod ip的8083端口去了。而且没有映射到主机上来,进入容器运行成功
[kafka@f3211e56f0ba ~]$ curl -H "Accept:application/json" http://172.17.0.5:8083/ {"version":"2.5.0","commit":"66563e712b0b9f84","kafka_cluster_id":"2r5ESXyUS_W1IvOmZVF3zg"}
2.部署mysql Connector
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 172.17.0.5:8083/connectors/ -d '{ "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.whitelist": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" } }'
这里dbserver1是指mysql server的逻辑识别id,这个值会作为kafka topic的前缀。(prefix)
3.查看变化
基于watch-topic工具查看各个队列中的情况
$ docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.2 watch-topic -a -k dbserver1.inventory.customers
不同的表的变化放到不同的队列中。
dbserver1.inventory.customers对应着mysql的customers表格
dbserver1.inventory.addresses对应着mysql的addresses表格
。。。。
因为变化是一串String可阅读性不强,比如更新的如下
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.customers.Key"},"payload":{"id":1004}} {"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"table"},{"type":"int64","optional":false,"field":"server_id"},{"type":"string","optional":true,"field":"gtid"},{"type":"string","optional":false,"field":"file"},{"type":"int64","optional":false,"field":"pos"},{"type":"int32","optional":false,"field":"row"},{"type":"int64","optional":true,"field":"thread"},{"type":"string","optional":true,"field":"query"}],"optional":false,"name":"io.debezium.connector.mysql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"dbserver1.inventory.customers.Envelope"},"payload":{"before":{"id":1004,"first_name":"Anne Marie","last_name":"Kretchmar","email":"annek@noanswer.org"},"after":{"id":1004,"first_name":"Anne","last_name":"Kretchmar","email":"annek@noanswer.org"},"source":{"version":"1.2.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1597310172000,"snapshot":"false","db":"inventory","table":"customers","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":1066,"row":0,"thread":9,"query":null},"op":"u","ts_ms":1597310172015,"transaction":null}}
可以通过在线的工具codebeautify进行查看,最主要关注payload部分。
https://codebeautify.org/jsonviewer
"payload": { "before": { "id": 1004, "first_name": "Anne Marie", "last_name": "Kretchmar", "email": "annek@noanswer.org" }, "after": { "id": 1004, "first_name": "Anne", "last_name": "Kretchmar", "email": "annek@noanswer.org" }, "source": { "version": "1.2.1.Final", "connector": "mysql", "name": "dbserver1", "ts_ms": 1597310172000, "snapshot": "false", "db": "inventory", "table": "customers", "server_id": 223344, "gtid": null, "file": "mysql-bin.000003", "pos": 1066, "row": 0, "thread": 9, "query": null }, "op": "u", "ts_ms": 1597310172015, "transaction": null }