DataX支持mysql8.X
Harvey2017 人气:1简介:
DataX 是一个异构数据源离线同步工具,致力于实现包括关系型数据库(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能。github地址: https://github.com/alibaba/DataX
1 注意部分
目前dataX不支持mysql8.X,需要修改源码,修改的地方
- OriginalConfPretreatmentUtil类中引用的DataBaseType的追加参数,mysql8的zeroDateTimeBehavior=convertToNull修改为zeroDateTimeBehavior=CONVERT_TO_NULL
修改前:suffix = "yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true";
修改后: suffix = "yearIsDateType=false&zeroDateTimeBehavior=CONVERT_TO_NULL&tinyInt1isBit=false&rewriteBatchedStatements=true";
- mysql驱动,在mysql reader和writer的pom文件修改为
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.11</version>
<https://img.qb5200.com/download-x/dependency>
- clean install 跳过测试
- 将reader和writer生成的target下面的datax的plugin拷贝到core工程项目和bin同级的plugin(源码生成是没有的,新建)
2 使用部分
- 目录级别
- json模板
{
"job": {
"setting": {
"speed": {
"byte":10485760
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column" : [
{
"value": "DataX",
"type": "string"
},
{
"value": 19890604,
"type": "long"
},
{
"value": "1989-06-04 00:00:00",
"type": "date"
},
{
"value": true,
"type": "bool"
},
{
"value": "test",
"type": "bytes"
}
],
"sliceRecordCount": 100000
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"print": false,
"encoding": "UTF-8"
}
}
}
]
}
}
- mysql示例json(github官网可以查看)
MysqlReader插件文档
MysqlWriter插件文档
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": ["id","name"],
"where": "id>0",
"connection": [
{
"table": [
"user"
],
"jdbcUrl": [
"jdbc:mysql://47.101.137.97:3306/test1?serverTimezone=UTC"
]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "root",
"password": "123456",
"column": ["id","name"],
"connection": [
{
"table": [
"user"
],
"jdbcUrl":"jdbc:mysql://47.101.137.97:3306/test2?serverTimezone=UTC"
}
]
}
}
}
],
"setting": {
"speed": {
"channel": 1,
"byte": 104857600
},
"errorLimit": {
"record": 10,
"percentage": 0.05
}
}
}
}
- 执行
进入到datax的bin目录(eg./Users/xuzhihui/test/backend/DataX-master/core/targethttps://img.qb5200.com/download-x/datax/bin),然后执行
python datax.py ../job/test.json
- 结果
加载全部内容