DataX进行数据同步总结

同步需求：mysql中的数据同步到atomdata（一个数据，和你认为的oracle一样）

Github主页地址：https://github.com/alibaba/DataX

一、dataX概览

1.1 DataX

DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、SQL Server、Oracle、PostgreSQL、HDFS、Hive、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

1.2 Features

DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。

1.3 System Requirements

• Linux
• JDK(1.8以上，推荐1.8)  
• Python(推荐Python2.6.X)
• Apache Maven 3.x (Compile DataX)

（1）jdk安装方法：

见链接： https://www.cnblogs.com/syw20170419/p/16250500.html

（2）python安装方法：

优先执行python -v， 如当前机器有python2.7版本，则跳过下面步骤

a、离线安装, 先将Python-2.7.15.zipPython-2.7.15.zip上传到工作机器

unzip Python-2.7.15.zip

cd Python-2.7.15
./configure --enable-optimizations

make altinstall

#修改系统变量

sudo vi  /etc/profile

#文件末尾增加一行内容

PATH=$PATH:/usr/src/Python-2.7.15

#保存退出

#验证是否安装成功，执行

python2 -v

（3）maven安装

见如下的二步骤中内容

二、安装dataX与应用

按照此文章的内容，进行安装，详见链接：https://developer.aliyun.com/article/216355

三、配置的json文件

（1）在服务器上创建并进入datax目录

cd /datax/job

（2）新建同步job文件

sudo vi job/mysq2atomdata-op.job

{
    "job": {
        "setting": {
            "speed": {
                "channel": 3000  #每次批量写入的条数
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [{
            "reader": {
                "name": "mysqlreader",
                "parameter": {
                    "username": "root",  #源数据库的用户名
                    "password": "123456",   #源数据库的密码
                    "column": ["hj_id","hj_name","hj_job","hj_jobAddress","hj_liveAddress","hj_mobile","hj_age","hj_column1","hj_column2","hj_column3","hj_column4","hj_column5","hj_column6","hj_column7","hj_column8","hj_column9","hj_column10","hj_column11","hj_column12","hj_column13","hj_column14","hj_column15","hj_column16","hj_column17","hj_column18","hj_column19","hj_column20","hj_column21","hj_column22","hj_column23"
                    ],  #需要筛选的列名称
                    "connection": [{
                        "table": ["hjry"],  #筛选的表
                        "jdbcUrl": ["jdbc:mysql://192.168.30.103:3306/syw_security"]  #连接的源地址/库名
                    }]
                }
            },
            "writer": {
                "name": "mysqlwriter",
                "parameter": {
                    "writeMode": "insert",
                    "username": "kepler",   #目标数据库的用户名
                    "password": "Aa123456",   #目标数据库的密码
                    "column": ["hj_id","hj_name","hj_job","hj_jobAddress","hj_liveAddress","hj_mobile","hj_age","hj_column1","hj_column2","hj_column3","hj_column4","hj_column5","hj_column6","hj_column7","hj_column8","hj_column9","hj_column10","hj_column11","hj_column12","hj_column13","hj_column14","hj_column15","hj_column16","hj_column17","hj_column18","hj_column19","hj_column20","hj_column21","hj_column22","hj_column23"
                    ],   #目标数据库的列名称
                    "session": ["set session sql_mode='ANSI'"],  
                    "preSql": ["delete from hjry"],   #hjry为目标数据库的表名称
                    "connection": [{
                        "jdbcUrl": "jdbc:mysql://192.168.30.103:3001/syw_security?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=UTC",  #目标数据库的配置
                        "table": ["hjry"]  #目标数据库的表
                    }]
                }
            }
        }]
    }
}

执行测试

python2  /datax/bin/datax.py /datax/job/hjry.job

即可看到执行结果

其他的json配置文件内容介绍可见链接：https://blog.csdn.net/qq_42764269/article/details/119577358

并发执行datax数据的导入：https://blog.csdn.net/mochou111/article/details/103570127