• Debezium初试


     向导地址

    https://debezium.io/documentation/reference/1.2/tutorial.html

    1.启动kafka connector

    $ docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka --link mysql:mysql debezium/connect:1.2

    按照文档说法,运行下面命令可见结果

    $ curl -H "Accept:application/json" localhost:8083/
    {"version":"2.5.0","commit":"cb8625948210849f"}

    但发现绑定8083不成功,定位半天发现connector绑定在pod ip的8083端口去了。而且没有映射到主机上来,进入容器运行成功

    [kafka@f3211e56f0ba ~]$ curl -H "Accept:application/json" http://172.17.0.5:8083/           
    {"version":"2.5.0","commit":"66563e712b0b9f84","kafka_cluster_id":"2r5ESXyUS_W1IvOmZVF3zg"}

    2.部署mysql Connector

    curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 172.17.0.5:8083/connectors/ -d '{ "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.whitelist": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" } }'

    这里dbserver1是指mysql server的逻辑识别id,这个值会作为kafka topic的前缀。(prefix)

    3.查看变化

    基于watch-topic工具查看各个队列中的情况

    $ docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.2 watch-topic -a -k dbserver1.inventory.customers

    不同的表的变化放到不同的队列中。

    dbserver1.inventory.customers对应着mysql的customers表格

    dbserver1.inventory.addresses对应着mysql的addresses表格

    。。。。

    因为变化是一串String可阅读性不强,比如更新的如下

    {"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.customers.Key"},"payload":{"id":1004}}    {"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"table"},{"type":"int64","optional":false,"field":"server_id"},{"type":"string","optional":true,"field":"gtid"},{"type":"string","optional":false,"field":"file"},{"type":"int64","optional":false,"field":"pos"},{"type":"int32","optional":false,"field":"row"},{"type":"int64","optional":true,"field":"thread"},{"type":"string","optional":true,"field":"query"}],"optional":false,"name":"io.debezium.connector.mysql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"dbserver1.inventory.customers.Envelope"},"payload":{"before":{"id":1004,"first_name":"Anne Marie","last_name":"Kretchmar","email":"annek@noanswer.org"},"after":{"id":1004,"first_name":"Anne","last_name":"Kretchmar","email":"annek@noanswer.org"},"source":{"version":"1.2.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1597310172000,"snapshot":"false","db":"inventory","table":"customers","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":1066,"row":0,"thread":9,"query":null},"op":"u","ts_ms":1597310172015,"transaction":null}}

    可以通过在线的工具codebeautify进行查看,最主要关注payload部分。

    https://codebeautify.org/jsonviewer

      "payload": {
        "before": {
          "id": 1004,
          "first_name": "Anne Marie",
          "last_name": "Kretchmar",
          "email": "annek@noanswer.org"
        },
        "after": {
          "id": 1004,
          "first_name": "Anne",
          "last_name": "Kretchmar",
          "email": "annek@noanswer.org"
        },
        "source": {
          "version": "1.2.1.Final",
          "connector": "mysql",
          "name": "dbserver1",
          "ts_ms": 1597310172000,
          "snapshot": "false",
          "db": "inventory",
          "table": "customers",
          "server_id": 223344,
          "gtid": null,
          "file": "mysql-bin.000003",
          "pos": 1066,
          "row": 0,
          "thread": 9,
          "query": null
        },
        "op": "u",
        "ts_ms": 1597310172015,
        "transaction": null
      }
  • 相关阅读:
    【2020Python修炼记】面向对象编程——绑定方法与非绑定方法
    【2020Python修炼记】面向对象编程——多态性与鸭子类型
    2020Python作业——封装2+继承
    2020Python作业——类与对象2+封装
    【2020Python修炼记】面向对象编程——继承与派生
    P1494 [国家集训队]小Z的袜子
    codeforces600E. Lomsat gelral(dsu on tree)
    这是个O2优化指令
    洛谷P1972 [SDOI2009]HH的项链
    poj3417
  • 原文地址:https://www.cnblogs.com/ericnie/p/13497892.html
Copyright © 2020-2023  润新知