canal是阿里巴巴旗下的一款开源项目,纯Java开发。基于数据库增量日志解析,提供增量数据订阅&消费,目前主要支持了MySQL(也支持mariaDB)。
起源:早期,阿里巴巴B2B公司因为存在杭州和美国双机房部署,存在跨机房同步的业务需求。不过早期的数据库同步业务,主要是基于trigger的方式获取增量变更,不过从2010年开始,阿里系公司开始逐步的尝试基于数据库的日志解析,获取增量变更进行同步,由此衍生出了增量订阅&消费的业务,从此开启了一段新纪元。
工作原理:mysql 的主从备份
伪装自己是mysql 的slave 解析binlog
开启mysql 的binlog
找到mysql的my.ini文件
打开文件后,添加以下内容
server_id=1 ###代表集群模式第一台机器 binlog_format=ROW ###行模式 log_bin=mysql_bin.log ###binlog的文件名称
执行sql语句
show variables like '%log_bin%'
查看是否开启binlog
create user canal identified by ‘canal’; grant select,replication slave,replication client on*.* to 'canal'@'%'; grant all privileges on . TO 'canal'@'%'; flush privileges
创建新的mysql用户 并赋予权限
下载canal服务https://github.com/alibaba/canal/releases/
打开canal服务conf下的配置文件canal.properties
这里配置端口号
复制conf目录下的example
粘贴到此目录下 修改文件名
进入你新建的文件夹 打开instance.properties
接着运行bin目录下的starup.bat
如果错误 删除starup.bat 中11行的@Rem
自此canal 的服务 就已经配置好了 接着需要搭建一个canal的客户端工程
package com.aila; import com.xpand.starter.canal.annotation.EnableCanalClient; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; /** * @Author: {---chenzhichao---} * @Date: 2020/6/2 15:11 */ @SpringBootApplication @EnableCanalClient public class MyCanalApplication { public static void main(String[] args) { SpringApplication.run(MyCanalApplication.class, args); } }
package com.aila.Listener; import com.alibaba.otter.canal.protocol.CanalEntry; import com.xpand.starter.canal.annotation.CanalEventListener; import com.xpand.starter.canal.annotation.ListenPoint; import org.springframework.amqp.rabbit.core.RabbitTemplate; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.core.StringRedisTemplate; import java.util.HashMap; import java.util.Map; import static com.alibaba.otter.canal.protocol.CanalEntry.EventType.DELETE; import static com.alibaba.otter.canal.protocol.CanalEntry.EventType.INSERT; import static com.alibaba.otter.canal.protocol.CanalEntry.EventType.UPDATE; /** * @Author: {---chenzhichao---} * @Date: 2020/6/8 11:00 */ @CanalEventListener public class NewsListener { @Autowired private RabbitTemplate rabbitTemplate; @Autowired private StringRedisTemplate stringRedisTemplate; @Autowired private RedisTemplate redisTemplate; @ListenPoint(schema = "class19",table = "city") public void mytest1(CanalEntry.EventType eventType, CanalEntry.RowData rowData){ Map<String,String> newData = new HashMap<>(); rowData.getAfterColumnsList().forEach((c)->newData.put(c.getName(),c.getValue())); for (String s : newData.keySet()) { System.out.println(s+":"+String.valueOf(newData.get(s))); } } @ListenPoint(schema = "class19",table = "hot_news") public void mytest2(CanalEntry.EventType eventType, CanalEntry.RowData rowData){ switch (eventType) { case INSERT: System.out.println("INSERT "); break; case UPDATE: System.out.println("UPDATE "); break; case DELETE: System.out.println("DELETE "); break; default: break; } } @ListenPoint(schema = "class19",table = "hot_news", eventType= {INSERT, UPDATE})//对热点信息表新增或更新 public void mytest3(CanalEntry.EventType eventType, CanalEntry.RowData rowData){ Map<String,String> newData = new HashMap<>(); rowData.getAfterColumnsList().forEach((c)->newData.put(c.getName(),c.getValue())); for (String s : newData.keySet()) { System.out.println(s+":"+String.valueOf(newData.get(s))); } String id = newData.get("id"); String content = newData.get("content"); String name = newData.get("name"); //stringRedisTemplate.boundValueOps(id).set(content); redisTemplate.boundHashOps(id+"hash").put(name,content); } @ListenPoint(schema = "class19",table = "hot_news",eventType= DELETE)//对热点信息表删除 public void mytest4(CanalEntry.EventType eventType, CanalEntry.RowData rowData){ Map<String,String> newData = new HashMap<>(); rowData.getBeforeColumnsList().forEach((c)->newData.put(c.getName(),c.getValue())); for (String s : newData.keySet()) { System.out.println(s+":"+String.valueOf(newData.get(s))); } String id = newData.get("id"); String content = newData.get("content"); String name = newData.get("name"); //stringRedisTemplate.delete(id); redisTemplate.delete(id+"hash"); } }
canal.client.instances.example.host=127.0.0.1 canal.client.instances.example.port=11111 canal.client.instances.example.batchSize=1000 spring.rabbitmq.host=127.0.0.1 spring.rabbitmq.username=czc spring.rabbitmq.password=qpalzm spring.redis.host=127.0.0.1 spring.datasource.url=jdbc:mysql://127.0.0.1:3306/class19?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
经过测试没有问题