• Impdp数据导入过程中exclude参数是否排除index的验证


      最近有个省级项目需要迁移大批量数据(大于2T;主要的表均为分区表,并且有二级分区,单表有129个分区,7-8个二级分区)并且迁移过程中需要转换字符集,从GBK转换到UTF-8。而且,涉及到新项目功能上线,迁移时间(停机时间)需要尽快能的短。数据字符集转换和导入时间尽量控制在10H以内。

          基于以上考虑,数据迁移和导入分一下几步操作:

          1.从现有的测试库上导出表结构数据,导入到目标库,除去索引和约束;没有索引和约束的额外开销,单纯导数据会很快。

          2.现有生产库上数据导出,尽可能采用高并发;考虑到新旧服务器CPU核数较多,准备采用parallel为32以上;

          3.数据分两步导入目标库:

                    3.1 第一步纯数据导入,即content=data_only;

                    3.2 第二步追加索引和约束,即include=index,constraint

      4.核对数据,表分析。

          以上主要步骤1和3分别验证如下:

    impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 exclude=index,constraint

    不会导入索引和约束

    impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics

    不会导入索引和约束;经过验证:TRIGGER,TABLE_STATISTICS 包含在metadata里面,建议去除trigger,statistics

    impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only

    索引和约束也会导入

    impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only

    仅导入表数据,不含index,constraint,trigger,statistics

    impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics

    追加index,constraint,trigger,statistics

    实验操作过程记录如下:


    [ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics

    Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:52:03 2021

    Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

    Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
    Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics
    Processing object type TABLE_EXPORT/TABLE/TABLE
    Job "SYSTEM"."20210311_ahxnb_ad71" successfully completed at Thu Mar 11 20:52:08 2021 elapsed 0 00:00:05

    [ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only

    Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:52:36 2021

    Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

    Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
    Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only
    Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
    . . imported "AHXNB"."AD71":"AAB301_341182"."AAB301_341182_P3" 11.20 MB 1334358 rows
    Job "SYSTEM"."20210311_ahxnb_ad71" successfully completed at Thu Mar 11 20:52:43 2021 elapsed 0 00:00:07

    [ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics

    Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:54:05 2021

    Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

    Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
    Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics
    Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
    Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
    Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
    Processing object type TABLE_EXPORT/TABLE/TRIGGER
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTBI_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTBI_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAI_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAI_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAU_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAU_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAD_AD71" created with compilation warnings
    ORA-39082: Object type TRIGGER:"AHXNB"."SBTAD_AD71" created with compilation warnings
    Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
    Job "SYSTEM"."20210311_ahxnb_ad71" completed with 8 error(s) at Thu Mar 11 20:54:40 2021 elapsed 0 00:00:34

      后续:

           采用该办法后,data_only导入大约3小时,追加索引和约束大约10小时。

            

            遇到的问题1:

      ORA-31693: Table data object "AHXNB"."AC63":"AAB301_341602"."AAB301_341602_P2014" failed to load/unload and is being skipped due to error:
      ORA-29913: error in executing ODCIEXTTABLEOPEN callout
      ORA-39077: unable to subscribe agent ORA-39077: unable to subscribe agent KUPC$A_1_163050148848000 to queue "KUPC$C_1_20210312150031"
      ORA-24067: exceeded maximum number of subscribers for queue SYS.KUPC$C_1_20210312150031

            后续发现该问题跟impdp导入过程中parallel参数有关系(过大?),后面通过配置parallel=1重新导入前面因出错未导入的数据。怀疑ORA-24067是一个BUG,留待后续继续观察测试。

            遇到的问题2:

            在解决问题1,补数据的时候遇到:

      ORA-31693: Table data object "AHXNB"."AC63":"AAB301_341602"."AAB301_341602_P2014" failed to load/unload and is being skipped due to error:
      ORA-04098: trigger 'AHXNB.SBTBI_AC63' is invalid and failed re-validation

            将该触发器disable后,数据顺利导入。后续重新enable该触发器。

  • 相关阅读:
    listener中@Autowired无法注入bean的一种解决方法
    PBKDF2加密的实现
    springmvc集成shiro后,session、request是否发生变化
    java加密解密工具类
    springMVC中controller层方法中使用private和public问题
    密码强度正则表达式 – 必须包含大写字母,小写字母和数字,至少8个字符等
    Java判断两个时间相差的天数
    Java中Date、String、Calendar类型之间的转化
    JavaScript前端和Java后端的AES加密和解密(转)
    JS前端加密JAVA后端解密详解
  • 原文地址:https://www.cnblogs.com/caoyibin/p/14520524.html
Copyright © 2020-2023  润新知