ETL工具之kettle的使用(一) MongoDB->Mysql

kettle简介：

　　Kettle是一款国外开源的ETL工具，纯java编写，可以在Window、Linux、Unix上运行，数据抽取高效稳定。

　　Kettle 中文名称叫水壶，该项目的主程序员MATT 希望把各种数据放到一个壶里，然后以一种指定的格式流出。

　　Kettle这个ETL工具集，它允许你管理来自不同数据库的数据，通过提供一个图形化的用户环境来描述你想做什么，而不是你想怎么做。

　　Kettle中有两种脚本文件，transformation和job，transformation完成针对数据的基础转换，job则完成整个工作流的控制

1：Kettle的下载与安装

　　下载地址 ： https://community.hds.com/docs/DOC-1009855

　　1) 下载下来是压缩版   解压就可以使用  解压后双击 Spoon.bat  等几秒 即出现图形界面工具

2：从MongoDB数据库的otherData文档 中 读取数据到 Mysql数据库t_other_data表 中

　　1)在kettle中，mongodb input组件。

　　

　　2) 编辑组件

　　3) 输入连接信息

　　4) 写查询语句

　　点击预览 可以预览成功说明配置无误

　　5) MongoDB->Mysql中MongoDB的配置就ok了    接下来配置表输出  即 mysql的配置

　　双击 表输出

 

　　6) 填写数据库连接信息

　　7) 点击测试 看mysql配置的是否无误 

　　看到successfully说明 mysql配置无误  (这里最初我配置的时候报错  报错信息后面单拎出来记录)

 　　8) 配置Mysql

　　9) 点击运行

　　10) 查看Mysql中t_other_data表中数据

 3：第一次在配置好Mysql后  点击测试 Mysql是报错的 

org.pentaho.di.core.exception.KettleDatabaseException: 
Error occurred while trying to connect to the database

Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed.
org.gjt.mm.mysql.Driver


    at org.pentaho.di.core.database.Database.normalConnect(Database.java:472)
    at org.pentaho.di.core.database.Database.connect(Database.java:370)
    at org.pentaho.di.core.database.Database.connect(Database.java:341)
    at org.pentaho.di.core.database.Database.connect(Database.java:331)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.createDatabaseNodes(XulDatabaseExplorerController.java:380)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.init(XulDatabaseExplorerController.java:130)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.initialize(AbstractXulDomContainer.java:141)
    at org.pentaho.ui.xul.swt.SwtXulRunner$1.run(SwtXulRunner.java:67)
    at org.eclipse.swt.widgets.Synchronizer.syncExec(Unknown Source)
    at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
    at org.pentaho.ui.xul.swt.SwtXulRunner.initialize(SwtXulRunner.java:64)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerDialog.open(XulDatabaseExplorerDialog.java:92)
    at org.pentaho.di.ui.core.database.dialog.DataOverrideHandler.explore(DataOverrideHandler.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
    at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
    at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
    at org.eclipse.jface.window.Window.open(Window.java:796)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:116)
    at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:61)
    at org.pentaho.di.ui.trans.step.BaseStepDialog.showDbDialogUnlessCancelledOrValid(BaseStepDialog.java:808)
    at org.pentaho.di.ui.trans.step.BaseStepDialog$EditConnectionListener.widgetSelected(BaseStepDialog.java:1515)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.trans.steps.tableoutput.TableOutputDialog.open(TableOutputDialog.java:888)
    at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
    at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8662)
    at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3293)
    at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1381)
    at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7817)
    at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9179)
    at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:707)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
Caused by: org.pentaho.di.core.exception.KettleDatabaseException: 
Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed.
org.gjt.mm.mysql.Driver

    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:515)
    at org.pentaho.di.core.database.Database.normalConnect(Database.java:456)
    ... 66 more
Caused by: java.lang.ClassNotFoundException: org.gjt.mm.mysql.Driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:490)
    ... 67 more

错误信息显示是jar包未安装

下载mysql-connector-java-5.0.7 jar包并放到lib目录下 关闭图形界面重新打开Spoon.bat即可

(有个小坑 ： 我第一次下载的版本是8依然报错，随后版本换成5好用了 )

下载jar包地址： http://mvnrepository.com/artifact/mysql/mysql-connector-java

思考：MongoDB的数据读取到Mysql中是手动处理的  这在生产环境合理吗？如何自动实时的读取数据呢？下节总结