ORACLE GOLDEN GATE oracle同步数据至kafka

一.服务器信息

ip		软件版本	ogg版本	软件包	操作系统版本	OGG安装路径
10.1.50.52	源	oracle11.2.0.4	12.2.0.1.1	V100692-01.zip	centos6.9	/data/ogg
10.0.71.102	目标	kafka0.10	12.2.0.1.160823	123010_ggs_Adapters_Linux_x64.zip	centos7.3	/data/ogg

二.源端安装OGG

1.解压V100692-01.zip

unzip V100692-01.zip

这个时候会看到fbo_ggs_Linux_x64_shiphome文件夹

2.我们采用静默安装的方式进行安装（注意要使用oracle用户来进行安装，oggcore.rsp配置需要根据实际需要进行修改，也可使用图形界面进行安装）

/usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent -responseFile /usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

三.源端OGG配置

1.开启主库归档日志、补充日志及force logging

alter database add supplemental log data;

alter database force logging;

alter system set enable_goldengate_replication=true scope=both;

2.创建ogg的管理用户

create user ogg identified by password account unlock;

grant connect,resource to ogg;

grant select any dictionary to ogg;

grant select any table to ogg;

grant execute on utl_file to ogg;

grant restricted session to ogg;

grant create table,create sequence to ogg;

alter table DCSDBA.YH_ITF_INVENTORY_TRANSACTION add supplemental log data (all) columns;--如果kafka需要接收到完整的修改信息就需要开启补全日志

3.登录ogg并创建目录

/data/ogg/ggsci

create subdirs

4.配置源端manager进程

edit params mgr

port 1357
autostart er *
autorestart er *

5.启动manager并确认状态

start mgr

info all 查看MANAGER进程为running状态

6.配置加密

ENCRYPT PASSWORD password ENCRYPTKEY DEFAULT

7.配置数据抽取进程

①加一个extract，名字随便取

add extract hnwmsprd,tranlog,begin now

②为这个ext指定他把远程文件放到远程机的哪个目录

ADD RMTTRAIL /data/ogg/dirdat/tt, EXTRACT hnwmsprd

③编辑配置文件

edit params hnwmsprd

extract hnwmsprd

setenv (NLS_LANG="AMERICAN_AMERICA.UTF8")

setenv (ORACLE_SID="hnwmsprd") 

userid ogg,password AACAAAAAAAAAAAKALILFJIXIECMGKFAHRBJHGDGBNJLBOAUC,ENCRYPTKEY DEFAULT

rmthost 10.0.71.102, mgrport 1357

rmttrail /data/ogg/dirdat/tt

discardfile /data/ogg/dirrpt/trail.dsc,append,megabytes 100

gettruncates

GETUPDATEBEFORES
NOCOMPRESSDELETES
NOCOMPRESSUPDATES

TRANLOGOPTIONS MINEFROMACTIVEDG

table DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

④启动进程

start hnwmsprd

四.OGG目标端搭建

目标端搭建只需解压软件即可

 

unzip 123010_ggs_Adapters_Linux_x64.zip

解压出来ggs_Adapters_Linux_x64.tar文件

将文件移动到/data/ogg目录下解压

mv ggs_Adapters_Linux_x64.tar /data/ogg
tar -xvf ggs_Adapters_Linux_x64.tar

五.OGG目标端配置

1.登录ogg并创建目录

/data/ogg/ggsci

create subdirs

2.配置源端manager进程

edit params mgr

port 1357
autostart er *
autorestart er *

3.启动manager并确认状态

start mgr

info all 查看MANAGER进程为running状态

4.添加复制进程

add replicat rkafka,exttrail dirdat/tt,begin now

edit params rkafka

REPLICAT rkafka

TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP DCSDBA.YH_ITF_INVENTORY_TRANSACTION, TARGET DCSDBA.YH_ITF_INVENTORY_TRANSACTION;

5.修改kafka.props（注意：kafka.props要放在/data/ogg/dirprm路径下）

gg.handlerlist = kafkahandler                                                                                                                           

gg.handler.kafkahandler.type = kafka                                                                                                             

gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties                     

gg.handler.kafkahandler.TopicName =hnwmsprd                                                                                          

gg.handler.kafkahandler.format =json                                                                                                           

gg.handler.kafkahandler.SchemaTopicName=ogg                                                                              

gg.handler.kafkahandler.BlockingSend =true                                                                                                

gg.handler.kafkahandler.includeTokens=false                                                                                               

gg.handler.kafkahandler.mode =tx                                                                                                                 

goldengate.userexit.timestamp=utc                                                                                                                  

goldengate.userexit.writers=javawriter                                                                                                            

javawriter.stats.display=TRUE                                                                                                                          

javawriter.stats.full=TRUE                                                                                                                                 

gg.log=log4j                                                                                                                                                          

gg.log.level=INFO                                                                                                                                               

gg.report.time=30sec                                                                                                                                          

gg.classpath=dirprm/:/usr/local/kafka/libs/*:                                                            

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar 

以上配置文件根据实际配置进行修改

6.编辑custom_kafka_producer.properties文件（注意：custom_kafka_producer.properties要放在/data/ogg/dirprm路径下）

#bootstrap.servers=host:port
bootstrap.servers=10.0.71.102:9092
acks=1
#compression.type=gzip
reconnect.backoff.ms=1000

value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=102400
linger.ms=10000

7.启动复制进程

start rkafka

六.验证

目标端执行

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hnwmsprd

源端更新DCSDBA.YH_ITF_INVENTORY_TRANSACTION表。

 如果是在adg作为源端需要在源抽取进程上添加参数

TRANLOGOPTIONS MINEFROMACTIVEDG  

并在主库开启ddl，可以参考下面这位大神的配置

http://czmmiao.iteye.com/blog/1609823