ORA-00600 3020 ORA-10567案例

 

PlateSpin克隆复制出的Oracle数据库服务器，往往启动数据库实例都会遇到一些杂七杂八的问题。今天测试DR环境时又遇到了一个特殊场景，在此之前，我已经遇到了下面两起案例：

ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [4194]

 

如下所示，在启动数据库实例时，遇到"ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []"错误

 

检查告警日志，就会看到下面详细信息（部分告警日志）：

Successful mount of redo thread 1, with mount id 4228022561
Tue Nov 22 15:11:33 CST 2016
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Tue Nov 22 15:11:33 CST 2016
ALTER DATABASE OPEN
Tue Nov 22 15:11:35 CST 2016
Beginning crash recovery of 1 threads
Tue Nov 22 15:11:35 CST 2016
Started redo scan
Tue Nov 22 15:11:35 CST 2016
Errors in file /u01/app/oracle/admin/SCM2/udump/scm2_ora_4465.trc:
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []
Tue Nov 22 15:11:36 CST 2016
Aborting crash recovery due to error 600
Tue Nov 22 15:11:36 CST 2016
Errors in file /u01/app/oracle/admin/SCM2/udump/scm2_ora_4465.trc:
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

 

遇到这个错误，大部分情况是出现在磁盘出现故障导致数据库崩溃后，实例启动失败(After a disk failure that caused the database to crash, the instance fails to start up with ORA-00600: arguments: [kcratr1_lastbwr].)，因为服务器是通过PlateSpin复制克隆过来的，在操作系统层面的复制，很难保证数据块的完全一致性，毕竟数据库服务器会频繁进行IO操作，除非关闭数据库实例，否则就有可能出现数据块不一致的情况,很有可能出现了写丢失，这种情况跟磁盘故障也有点类似（当然具体原理不是很清楚）。此时我只能关闭数据库实例，然后startup mount后，recover database。如下所示

 

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 3.4360E+10 bytes
Fixed Size                  2159376 bytes
Variable Size            2.4931E+10 bytes
Database Buffers         9378463744 bytes
Redo Buffers               48168960 bytes
Database mounted.
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [52], [280690], [218384498],
[], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280690)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'


SQL>

 

结果恢复时又遇到了ORA-00600: internal error code, arguments: [3020] 与 ORA-10567: Redo is inconsistent with data block 错误。

PMON started with pid=2, OS id=4508
PSP0 started with pid=3, OS id=4510
MMAN started with pid=4, OS id=4512
DBW0 started with pid=5, OS id=4514
LGWR started with pid=6, OS id=4516
CKPT started with pid=7, OS id=4518
SMON started with pid=8, OS id=4520
RECO started with pid=9, OS id=4522
CJQ0 started with pid=10, OS id=4524
MMON started with pid=11, OS id=4526
Tue Nov 22 15:14:52 CST 2016
starting up 8 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=4528
Tue Nov 22 15:14:53 CST 2016
starting up 12 shared server(s) ...
Oracle Data Guard is not available in this edition of Oracle.
Tue Nov 22 15:14:53 CST 2016
ALTER DATABASE   MOUNT
Tue Nov 22 15:14:59 CST 2016
Setting recovery target incarnation to 5
Tue Nov 22 15:14:59 CST 2016
Successful mount of redo thread 1, with mount id 4228071407
Tue Nov 22 15:14:59 CST 2016
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Tue Nov 22 15:15:15 CST 2016
ALTER DATABASE RECOVER  database
Tue Nov 22 15:15:15 CST 2016
Media Recovery Start
Tue Nov 22 15:15:15 CST 2016
Recovery of Online Redo Log: Thread 1 Group 6 Seq 64605 Reading mem 0
  Mem# 0: /u01/oradata/SCM2/redo06.log
  Mem# 1: /u03/oradata/SCM2/redo06.log
Tue Nov 22 15:15:21 CST 2016
Errors in file /u01/app/oracle/admin/SCM2/udump/scm2_ora_4570.trc:
ORA-00600: internal error code, arguments: [3020], [52], [280690], [218384498], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280690)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'
Tue Nov 22 15:15:22 CST 2016
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

 

出现这个错误后，恢复（Recover）中断，无法继续，这个是因为恢复时应用Redo log时发现Redo里面的记录的信息与被恢复的数据块信息不一致，导致恢复无法继续。官方文档介绍如下：

 

CAUSE

Recovery stops because of failed consistency checks, a problem called stuck recovery. Stuck recovery can occur when an underlying operating system or storage system loses a write issued by the database during normal operation. There is an inconsistency between the information stored in the redo and the information stored in a database block being recovered.

The database signals an internal error when applying the redo. This problem can be caused by an Oracle Database bug or may be because of I/O problem ( hardware or O/S related issue )

There is a known EMC issue related to an RDBMS ORA-600 [3020] where the root-cause is on OS/Hardware level.

Details from EMC on the nature of the fix (problem with Symmetrix microcode)

     ID: emc230687

     Domain: EMC1

    Solution Class: 3.X Compatibility

SOLUTION

When media recovery encounters a problem, the alert log may indicate that recovery can continue if it is allowed to corrupt the data block causing the problem. The alert log contains information about the block: its block type, block address, the tablespace it belongs to, and so forth. For blocks containing user data, the alert log may also report the data object number.

In this case, the database can proceed with recovery if it is allowed to mark the problem block as corrupt. Nevertheless, this response is not always advisable. For example, if the block is an important block in the SYSTEM tablespace, marking the block as corrupt can eventually prevent you from opening the recovered database. Another consideration is whether the recovery problem is isolated. If this problem is followed immediately by many other problems in the redo stream, then you may want to open the database with the RESETLOGS option.

For a block containing user data, you can usually query the database to find out which object or table owns this block. If the database is not open, then you should be able to open the database read-only, even if you are recovering a whole database backup. The following example cancels recovery and opens read-only:

…………………………………………………………………………………………………………………………………………………………………………………………………………………

 

尝试了一下能否以read only打开数据库，结果报ORA-16005: database requires recovery错误。检查数据文件和控制文件，发现checkpoint都是一致的。

SQL> alter database open read only;
alter database open read only
*
ERROR at line 1:
ORA-16005: database requires recovery


SQL> col checkpoint_change# for 9999999999999999
SQL> select checkpoint_change# from v$database;

CHECKPOINT_CHANGE#
------------------
       23493915876

SQL> select file#, nvl(last_change#,0),checkpoint_change#
  2  from v$datafile
  3  where file#=52;

     FILE# NVL(LAST_CHANGE#,0) CHECKPOINT_CHANGE#
---------- ------------------- ------------------
        52                   0        23493915876

SQL>

 

然后我就按照官方文档使用recover database allow 1 corruption 将不一致的块标记为损坏，然后可以执行进一步的恢复尝试来恢复数据库。恢复过程中，发现不一致性的数据块数量不止一个，反复执行下面命令，终于熬到了Media recovery complete了

SQL> startup mount;
ORACLE instance started.

Total System Global Area 3.4360E+10 bytes
Fixed Size                  2159376 bytes
Variable Size            2.4931E+10 bytes
Database Buffers         9378463744 bytes
Redo Buffers               48168960 bytes
Database mounted.
SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [52], [280688], [218384496], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280688)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'


SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [39], [95934], [163673790], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 39, block# 95934)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 39: '/u02/oradata/SCM2/escmowner_d02.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 358585


SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [39], [95936], [163673792], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 39, block# 95936)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 39: '/u02/oradata/SCM2/escmowner_d02.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 358585


SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [39], [95936], [163673792], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 39, block# 95936)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 39: '/u02/oradata/SCM2/escmowner_d02.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 358585


SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [52], [280689], [218384497], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280689)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'


SQL> recover database allow 10 corruption;
ORA-10588: Can only allow 1 corruption for normal media/standby recovery


SQL> recover database allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [73], [1639], [306185831], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 73, block# 1639)
ORA-10564: tablespace ESCMOWNER_IDX
ORA-01110: data file 73: '/u03/oradata/SCM2/escmowner_x10.dbf'
ORA-10560: block type '0'


SQL> recover database allow 1 corruption;
Media recovery complete.
SQL>

 

操作过程中对应的部分告警日志内容如下所示：

 

ALTER DATABASE RECOVER  database allow 1 corruption
Wed Nov 23 09:42:11 CST 2016
Media Recovery Start
 ALLOW CORRUPTION option must use serial recovery
Wed Nov 23 09:42:11 CST 2016
Recovery of Online Redo Log: Thread 1 Group 6 Seq 64605 Reading mem 0
  Mem# 0: /u01/oradata/SCM2/redo06.log
  Mem# 1: /u03/oradata/SCM2/redo06.log
CORRUPTING BLOCK 280690 OF FILE 52 AND CONTINUING RECOVERY
Wed Nov 23 09:42:12 CST 2016
Errors in file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc:
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280690)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'
Hex dump of (file 64, block 290183) in trace file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc
Corrupt block relative dba: 0x10046d87 (file 64, block 290183)
Fractured block found during media recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x10046d87
 last change scn: 0x0005.785a119c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x9d890601
 check value in block header: 0x3110
 computed block checksum: 0x8e14
Reread of rdba: 0x10046d87 (file 64, block 290183) found same corrupted data
Hex dump of (file 52, block 280687) in trace file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc
Corrupt block relative dba: 0x0d04486f (file 52, block 280687)
Fractured block found during media recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x0d04486f
 last change scn: 0x0005.785a119c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xac6e1b02
 check value in block header: 0xf67b
 computed block checksum: 0x17c1
Reread of rdba: 0x0d04486f (file 52, block 280687) found same corrupted data
Wed Nov 23 09:42:25 CST 2016
Errors in file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc:
ORA-00600: internal error code, arguments: [3020], [52], [280688], [218384496], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 52, block# 280688)
ORA-10564: tablespace ESCMOWNER_DATA
ORA-01110: data file 52: '/u02/oradata/SCM2/escmowner_d07.dbf'
ORA-10560: block type 'LOB BLOCK'
Wed Nov 23 09:42:25 CST 2016
Hex dump of (file 52, block 280687) in trace file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc
Corrupt block relative dba: 0x0d04486f (file 52, block 280687)
Fractured block found during in-flux buffer recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x0d04486f
 last change scn: 0x0005.785a119c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xac6e1b02
 check value in block header: 0xf67b
 computed block checksum: 0x17c1
Reread of rdba: 0x0d04486f (file 52, block 280687) found same corrupted data
Hex dump of (file 64, block 290183) in trace file /u01/app/oracle/admin/SCM2/udump/scm2_ora_20164.trc
Corrupt block relative dba: 0x10046d87 (file 64, block 290183)
Fractured block found during in-flux buffer recovery
Data in bad block:
 type: 6 format: 2 rdba: 0x10046d87
 last change scn: 0x0005.785a119c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x9d890601
 check value in block header: 0x3110
 computed block checksum: 0x8e14
Reread of rdba: 0x10046d87 (file 64, block 290183) found same corrupted data
Wed Nov 23 09:42:25 CST 2016
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  database allow 1 corruption  ...

 

但是在打开数据库的时候，又遇到了ORA-01578与ORA-01110错误，如下所示：

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 98)
ORA-01110: data file 1: '/u01/oradata/SCM2/system01.dbf'

 

出现ORA-01578，意味着数据库出现了坏块，使用dbv命令检查数据库坏块，发现标记为坏块的有32个之多。

dbv file=/u01/oradata/SCM2/system01.dbf blocksize=8192

.............................................................

Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x0040f1ca
 last change scn: 0x0005.781d41ff seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x1daf0601
 check value in block header: 0x9cf8
 computed block checksum: 0x5e57

Page 63922 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040f9b2 (file 1, block 63922)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x0040f9b2
 last change scn: 0x0005.78210b86 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x4eff0601
 check value in block header: 0x8642
 computed block checksum: 0x47b4

Page 64336 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040fb50 (file 1, block 64336)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x0040fb50
 last change scn: 0x0005.780c413f seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x7a550601
 check value in block header: 0x9e5f
 computed block checksum: 0x46a

Page 73356 is influx - most likely media corrupt
Corrupt block relative dba: 0x00411e8c (file 1, block 73356)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00411e8c
 last change scn: 0x0005.75c8101d seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x18b70601
 check value in block header: 0x964a
 computed block checksum: 0x8aa

Page 73902 is influx - most likely media corrupt
Corrupt block relative dba: 0x004120ae (file 1, block 73902)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x004120ae
 last change scn: 0x0005.77a57f68 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xfa4d0601
 check value in block header: 0x729d
 computed block checksum: 0xdf7f

Page 75400 is influx - most likely media corrupt
Corrupt block relative dba: 0x00412688 (file 1, block 75400)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00412688
 last change scn: 0x0005.76ce99af seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xc0ad0601
 check value in block header: 0x25b
 computed block checksum: 0x5ea1

Page 75448 is influx - most likely media corrupt
Corrupt block relative dba: 0x004126b8 (file 1, block 75448)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x004126b8
 last change scn: 0x0005.78211186 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x72500601
 check value in block header: 0x5969
 computed block checksum: 0x63d6

Page 75512 is influx - most likely media corrupt
Corrupt block relative dba: 0x004126f8 (file 1, block 75512)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x004126f8
 last change scn: 0x0005.78579e34 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x52460601
 check value in block header: 0x17b2
 computed block checksum: 0xda45

Page 75830 is influx - most likely media corrupt
Corrupt block relative dba: 0x00412836 (file 1, block 75830)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00412836
 last change scn: 0x0005.78211186 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x1cf10601
 check value in block header: 0x5455
 computed block checksum: 0x1f7a

Page 79066 is influx - most likely media corrupt
Corrupt block relative dba: 0x004134da (file 1, block 79066)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x004134da
 last change scn: 0x0005.78548bf1 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xf0150601
 check value in block header: 0x29aa
 computed block checksum: 0x7ada

Page 79416 is influx - most likely media corrupt
Corrupt block relative dba: 0x00413638 (file 1, block 79416)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00413638
 last change scn: 0x0005.782111d1 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x1b610601
 check value in block header: 0xd763
 computed block checksum: 0xbf6e

Page 80102 is influx - most likely media corrupt
Corrupt block relative dba: 0x004138e6 (file 1, block 80102)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x004138e6
 last change scn: 0x0005.78196cc8 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x8a210601
 check value in block header: 0xe562
 computed block checksum: 0xe6e9

Page 80420 is influx - most likely media corrupt
Corrupt block relative dba: 0x00413a24 (file 1, block 80420)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00413a24
 last change scn: 0x0005.77bd6fba seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x2bf80602
 check value in block header: 0xadf
 computed block checksum: 0x5e98

Page 80930 is influx - most likely media corrupt
Corrupt block relative dba: 0x00413c22 (file 1, block 80930)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00413c22
 last change scn: 0x0005.77302b26 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x2ae30601
 check value in block header: 0x21ad
 computed block checksum: 0x4ae6

Page 81250 is influx - most likely media corrupt
Corrupt block relative dba: 0x00413d62 (file 1, block 81250)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00413d62
 last change scn: 0x0005.7821123c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x7fba0602
 check value in block header: 0xf36
 computed block checksum: 0x42a6

Page 81472 is influx - most likely media corrupt
Corrupt block relative dba: 0x00413e40 (file 1, block 81472)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00413e40
 last change scn: 0x0005.7847cd31 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x8cdb0601
 check value in block header: 0xeb69
 computed block checksum: 0x41ea

Page 85757 is influx - most likely media corrupt
Corrupt block relative dba: 0x00414efd (file 1, block 85757)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00414efd
 last change scn: 0x0005.73cab1bc seq: 0x2 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x4a100601
 check value in block header: 0xf053
 computed block checksum: 0x700

Page 85996 is influx - most likely media corrupt
Corrupt block relative dba: 0x00414fec (file 1, block 85996)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00414fec
 last change scn: 0x0005.78210a33 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xe96e0601
 check value in block header: 0x1880
 computed block checksum: 0xe35c

Page 87348 is influx - most likely media corrupt
Corrupt block relative dba: 0x00415534 (file 1, block 87348)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00415534
 last change scn: 0x0005.781f1f02 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x2cfd0602
 check value in block header: 0xb8ae
 computed block checksum: 0x596



DBVERIFY - Verification complete

Total Pages Examined         : 88320
Total Pages Processed (Data) : 53676
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 16588
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 3306
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14718
Total Pages Marked Corrupt   : 32
Total Pages Influx           : 32
Highest block SCN            : 2019287292 (5.2019287292)

 

看来只能用备份（本机的备份）恢复来修复坏块，结果才想起这个实例时标准版，无法使用这个功能。

 

RMAN> blockrecover datafile 1 block 63922;

Starting blockrecover at 23-NOV-16
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of blockrecover command at 11/23/2016 12:24:41
RMAN-05009: Block Media Recovery requires Enterprise Edition

 

使用RMAN还原备份文件结果遭遇ORA-19870与ORA-19599错误

RMAN> restore datafile 1;

Starting restore at 23-NOV-16
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /u01/oradata/SCM2/system01.dbf
channel ORA_DISK_1: reading from backup piece /u05/backup/backupsets/ora_df928290157_s27337_s1
ORA-19870: error reading backup piece /u05/backup/backupsets/ora_df928290157_s27337_s1
ORA-19599: block number 39555 is corrupt in backup piece /u05/backup/backupsets/ora_df928290157_s27337_s1
failover to previous backup

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 11/23/2016 14:07:56
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 1 found to restore

 

出现ORA-19599错误，意味着备份集文件也出现了坏块。如下描述所示，只能通过拷贝生产环境的正常的备份集来恢复了。折腾了这么久，对PlateSpin这个DR方案颇感失望。

Error: ORA-19599 block number %s is corrupt in %s %s

----------------------------------------------------------------------------

Cause: A corrupt block was found in a control file, archivelog, or backup

piece that is being read for a backup or copy. Corruption shall not be

tolerated in control files, archivelogs, or backup pieces.

Action: None. The copy or backup operation fails. Note that in the case of a

backup set, the conversation is still active and the piece may be

retried.

 

参考资料：

 

http://www.eygle.com/archives/2010/03/ora_00600_3020.html

Stuck recovery of database ORA-00600[3020] (文档 ID 283269.1)

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=346216290638124&parent=DOCUMENT&sourceId=1088018.1&id=28814.1&_afrWindowMode=0&_adf.ctrl-state=15uednnal0_132