oracle数据库启动报错，不能启动ASM实例

数据库rac启动时报错，日志例如以下，后来使用

Sat Jun  7 06:02:11 2014
GATHER_STATS_JOB encountered errors.  Check the trace file.
Sat Jun  7 06:02:11 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-08103: object no longer exists
Sat Jun  7 06:02:51 2014
Thread 2 advanced to log sequence 13292 (LGWR switch)
  Current log# 4 seq# 13292 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun  7 06:32:10 2014
Trace dumping is performing id=[cdmp_20140607063209]
Sat Jun  7 07:01:33 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j000_6797.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 07:01:35 2014
Trace dumping is performing id=[cdmp_20140607070135]
Sat Jun  7 07:39:01 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_24893.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 07:39:04 2014
Trace dumping is performing id=[cdmp_20140607073904]
Sat Jun  7 08:00:42 2014
Trace dumping is performing id=[cdmp_20140607080042]
Sat Jun  7 08:30:13 2014
Trace dumping is performing id=[cdmp_20140607083013]
Sat Jun  7 09:00:43 2014
Trace dumping is performing id=[cdmp_20140607090043]
Sat Jun  7 09:36:24 2014
Trace dumping is performing id=[cdmp_20140607093623]
Sat Jun  7 10:00:46 2014
Trace dumping is performing id=[cdmp_20140607100046]
Sat Jun  7 10:38:31 2014
Trace dumping is performing id=[cdmp_20140607103831]
Sat Jun  7 11:00:52 2014
Trace dumping is performing id=[cdmp_20140607110051]
Sat Jun  7 11:37:40 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_18978.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 11:37:43 2014
Trace dumping is performing id=[cdmp_20140607113743]
Sat Jun  7 12:00:51 2014
Trace dumping is performing id=[cdmp_20140607120050]
Sat Jun  7 12:38:58 2014
Trace dumping is performing id=[cdmp_20140607123858]
Sat Jun  7 13:00:56 2014
Trace dumping is performing id=[cdmp_20140607130056]
Sat Jun  7 13:39:25 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_31749.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 13:39:28 2014
Trace dumping is performing id=[cdmp_20140607133928]
Sat Jun  7 13:44:08 2014
Thread 2 advanced to log sequence 13293 (LGWR switch)
  Current log# 3 seq# 13293 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun  7 14:00:59 2014
Trace dumping is performing id=[cdmp_20140607140058]
Sat Jun  7 14:38:47 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_5659.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 14:38:48 2014
Trace dumping is performing id=[cdmp_20140607143848]
Sat Jun  7 15:00:44 2014
Trace dumping is performing id=[cdmp_20140607150043]
Sat Jun  7 15:34:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_12186.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 15:34:05 2014
Trace dumping is performing id=[cdmp_20140607153405]
Sat Jun  7 15:43:52 2014
Thread 2 advanced to log sequence 13294 (LGWR switch)
  Current log# 4 seq# 13294 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun  7 16:02:14 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 16:02:19 2014
Trace dumping is performing id=[cdmp_20140607160219]
Sat Jun  7 16:03:35 2014
Thread 2 advanced to log sequence 13295 (LGWR switch)
  Current log# 3 seq# 13295 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun  7 16:40:00 2014
Trace dumping is performing id=[cdmp_20140607164000]
Sat Jun  7 17:01:30 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 17:01:33 2014
Trace dumping is performing id=[cdmp_20140607170133]
Sat Jun  7 17:32:37 2014
Trace dumping is performing id=[cdmp_20140607173236]
Sat Jun  7 18:01:51 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 18:01:53 2014
Trace dumping is performing id=[cdmp_20140607180153]
Sat Jun  7 18:29:15 2014
Thread 2 advanced to log sequence 13296 (LGWR switch)
  Current log# 4 seq# 13296 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sat Jun  7 18:30:52 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19302.trc:
Sat Jun  7 18:30:54 2014
Trace dumping is performing id=[cdmp_20140607183054]
Sat Jun  7 18:30:58 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19300.trc:
Sat Jun  7 18:31:05 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_19300.trc:
Sat Jun  7 18:42:14 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_31207.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 18:42:20 2014
Trace dumping is performing id=[cdmp_20140607184220]
Sat Jun  7 19:00:04 2014
Trace dumping is performing id=[cdmp_20140607190003]
Sat Jun  7 19:31:17 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_4965.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 19:31:18 2014
Trace dumping is performing id=[cdmp_20140607193118]
Sat Jun  7 20:00:42 2014
Trace dumping is performing id=[cdmp_20140607200042]
Sat Jun  7 20:31:30 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_11315.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sat Jun  7 20:31:32 2014
Trace dumping is performing id=[cdmp_20140607203132]
Sat Jun  7 21:33:02 2014
Trace dumping is performing id=[cdmp_20140607213302]
Sat Jun  7 22:02:16 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j000_6797.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 22:17:00 2014
Thread 2 advanced to log sequence 13297 (LGWR switch)
  Current log# 3 seq# 13297 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sat Jun  7 22:30:38 2014
Trace dumping is performing id=[cdmp_20140607223037]
Sat Jun  7 23:01:21 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_13352.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Sat Jun  7 23:01:25 2014
Trace dumping is performing id=[cdmp_20140607230125]
Sat Jun  7 23:30:33 2014
Trace dumping is performing id=[cdmp_20140607233032]
Sun Jun  8 00:00:38 2014
Trace dumping is performing id=[cdmp_20140608000038]
Sun Jun  8 00:39:30 2014
Trace dumping is performing id=[cdmp_20140608003930]
Sun Jun  8 01:00:36 2014
Trace dumping is performing id=[cdmp_20140608010036]
Sun Jun  8 01:45:14 2014
Trace dumping is performing id=[cdmp_20140608014514]
Sun Jun  8 02:00:41 2014
Trace dumping is performing id=[cdmp_20140608020041]
Sun Jun  8 02:36:03 2014
Trace dumping is performing id=[cdmp_20140608023603]
Sun Jun  8 03:31:51 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_24887.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 03:31:54 2014
Trace dumping is performing id=[cdmp_20140608033154]
Sun Jun  8 04:05:37 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:15 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 04:47:20 2014
Trace dumping is performing id=[cdmp_20140608044719]
Sun Jun  8 05:00:22 2014
Trace dumping is performing id=[cdmp_20140608050022]
Sun Jun  8 05:39:23 2014
Trace dumping is performing id=[cdmp_20140608053922]
Sun Jun  8 06:00:28 2014
Trace dumping is performing id=[cdmp_20140608060027]
Sun Jun  8 06:32:45 2014
Trace dumping is performing id=[cdmp_20140608063245]
Sun Jun  8 07:00:30 2014
Trace dumping is performing id=[cdmp_20140608070030]
Sun Jun  8 07:11:17 2014
Thread 2 advanced to log sequence 13298 (LGWR switch)
  Current log# 4 seq# 13298 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Sun Jun  8 07:30:12 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_6955.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 07:30:15 2014
Trace dumping is performing id=[cdmp_20140608073015]
Sun Jun  8 08:00:32 2014
Trace dumping is performing id=[cdmp_20140608080031]
Sun Jun  8 08:30:04 2014
Trace dumping is performing id=[cdmp_20140608083004]
Sun Jun  8 09:36:04 2014
Trace dumping is performing id=[cdmp_20140608093604]
Sun Jun  8 10:00:27 2014
Trace dumping is performing id=[cdmp_20140608100026]
Sun Jun  8 10:35:29 2014
Trace dumping is performing id=[cdmp_20140608103528]
Sun Jun  8 11:00:32 2014
Trace dumping is performing id=[cdmp_20140608110031]
Sun Jun  8 11:31:07 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_32192.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 11:31:13 2014
Trace dumping is performing id=[cdmp_20140608113113]
Sun Jun  8 12:39:37 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_5862.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 12:39:38 2014
Trace dumping is performing id=[cdmp_20140608123938]
Sun Jun  8 13:00:20 2014
Trace dumping is performing id=[cdmp_20140608130020]
Sun Jun  8 13:40:14 2014
Trace dumping is performing id=[cdmp_20140608134014]
Sun Jun  8 14:00:21 2014
Trace dumping is performing id=[cdmp_20140608140020]
Sun Jun  8 14:35:35 2014
Trace dumping is performing id=[cdmp_20140608143534]
Sun Jun  8 15:34:51 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Sun Jun  8 15:37:47 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_25433.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 15:37:50 2014
Trace dumping is performing id=[cdmp_20140608153750]
Sun Jun  8 16:13:50 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=121
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_11844.trc
Sun Jun  8 16:58:04 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=136
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_25750.trc
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:19 2014
WARNING: inbound connection timed out (ORA-3136)
Sun Jun  8 21:36:21 2014
Trace dumping is performing id=[cdmp_20140608213621]
Sun Jun  8 21:36:22 2014
Thread 2 advanced to log sequence 13299 (LGWR switch)
  Current log# 3 seq# 13299 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Sun Jun  8 22:00:15 2014
Trace dumping is performing id=[cdmp_20140608220015]
Sun Jun  8 22:40:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_22234.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 22:40:06 2014
Trace dumping is performing id=[cdmp_20140608224006]
Sun Jun  8 23:00:18 2014
Trace dumping is performing id=[cdmp_20140608230018]
Sun Jun  8 23:29:52 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_27653.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Sun Jun  8 23:29:53 2014
Trace dumping is performing id=[cdmp_20140608232953]
Mon Jun  9 00:44:11 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_30862.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []
Mon Jun  9 00:44:15 2014
Trace dumping is performing id=[cdmp_20140609004415]
Mon Jun  9 01:24:41 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=174
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_13953.trc
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:22 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Jun  9 05:56:24 2014
Trace dumping is performing id=[cdmp_20140609055624]
Mon Jun  9 05:57:37 2014
Thread 2 advanced to log sequence 13300 (LGWR switch)
  Current log# 4 seq# 13300 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Mon Jun  9 06:07:55 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j006_15568.trc:
ORA-12012: error on auto execute of job 524
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at "ZDDX_NEW.PKG_BB_LKZHFJTJ_W", line 537
ORA-06512: at "ZDDX_NEW.PKG_BB_LKZHFJTJ_W", line 9
ORA-06512: at "ZDDX_NEW.PKG_BB_TOOL_W", line 309
ORA-06512: at line 1
Mon Jun  9 06:41:35 2014
Thread 2 advanced to log sequence 13301 (LGWR switch)
  Current log# 3 seq# 13301 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Mon Jun  9 06:57:53 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=54
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_mmon_1398.trc
Mon Jun  9 07:24:19 2014
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=174
System State dumped to trace file /oracle/product/admin/dqb/udump/dqb2_ora_1963.trc
Mon Jun  9 09:45:09 2014
Trace dumping is performing id=[cdmp_20140609094508]
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_asmb_1428.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:36 2014
ASMB: terminating instance due to error 15064
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lmon_1337.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_pmon_1331.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms5_1361.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms1_1345.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms2_1349.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms6_1365.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms0_1341.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms7_1369.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms3_1353.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms4_1357.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:36 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lmd0_1339.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:37 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_1545.trc:
ORA-00600: internal error code, arguments: [kfnDisconnect], [not connected], [], [], [], [], [], []
ORA-15055: unable to connect to ASM instance
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-03113: end-of-file on communication channel
Mon Jun  9 09:45:38 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_rbal_1432.trc:
ORA-15064: communication failure with ASM instance
Mon Jun  9 09:45:39 2014
Trace dumping is performing id=[cdmp_20140609094539]
Mon Jun  9 09:45:47 2014
Instance terminated by ASMB, pid = 1428
Mon Jun  9 09:53:01 2014
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.177.1.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 300
  sessions                 = 335
  __shared_pool_size       = 452984832
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  spfile                   = +DG/dqb/spfiledqb.ora
  sga_target               = 1610612736
  control_files            = +DG/dqb/controlfile/current.272.803844991
  control_file_record_keep_time= 8
  db_block_size            = 8192
  __db_cache_size          = 1107296256
  compatible               = 10.2.0.3.0
  log_archive_dest_1       = LOCATION=+dg/dqb/arch/arch1
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DG
  db_recovery_file_dest_size= 10737418240
  thread                   = 2
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS4
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=dqbXDB)
  remote_listener          = LISTENERS_DQB
  job_queue_processes      = 10
  background_dump_dest     = /oracle/product/admin/dqb/bdump
  user_dump_dest           = /oracle/product/admin/dqb/udump
  core_dump_dest           = /oracle/product/admin/dqb/cdump
  audit_file_dest          = /oracle/product/admin/dqb/adump
  db_name                  = dqb
  open_cursors             = 300
  pga_aggregate_target     = 3363831808
Cluster communication is configured to use the following interface(s) for this instance
  192.168.1.2
Mon Jun  9 09:53:01 2014
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=4, OS id=13866
DIAG started with pid=8, OS id=13868
PSP0 started with pid=12, OS id=13870
LMON started with pid=16, OS id=13872
LMD0 started with pid=2, OS id=13874
LMS0 started with pid=6, OS id=13876
LMS1 started with pid=3, OS id=13880
LMS2 started with pid=20, OS id=13884
LMS3 started with pid=5, OS id=13888
LMS4 started with pid=10, OS id=13892
LMS5 started with pid=7, OS id=13896
LMS6 started with pid=24, OS id=13900
LMS7 started with pid=9, OS id=13904
MMAN started with pid=28, OS id=13908
DBW0 started with pid=14, OS id=13910
DBW1 started with pid=11, OS id=13912
DBW2 started with pid=32, OS id=13914
DBW3 started with pid=13, OS id=13916
LGWR started with pid=15, OS id=13918
CKPT started with pid=36, OS id=13920
SMON started with pid=40, OS id=13922
RECO started with pid=44, OS id=13924
CJQ0 started with pid=18, OS id=13926
MMON started with pid=48, OS id=13928
Mon Jun  9 09:53:02 2014
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=52, OS id=13936
Mon Jun  9 09:53:02 2014
starting up 1 shared server(s) ...
Mon Jun  9 09:53:02 2014
lmon registered with NM - instance id 2 (internal mem no 1)
Mon Jun  9 09:53:02 2014
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Mon Jun  9 09:53:02 2014
 LMS 2: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 0: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 3: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 6: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 7: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 5: 0 GCS shadows cancelled, 0 closed
Mon Jun  9 09:53:02 2014
 LMS 4: 0 GCS shadows cancelled, 0 closed
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Mon Jun  9 09:53:02 2014
 LMS 2: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 3: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 7: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 5: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 4: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 LMS 6: 0 GCS shadows traversed, 0 replayed
Mon Jun  9 09:53:02 2014
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=30, OS id=13942
Mon Jun  9 09:53:03 2014
ALTER DATABASE   MOUNT
Mon Jun  9 09:53:03 2014
This instance was first to mount
Mon Jun  9 09:53:03 2014
Starting background process ASMB
ASMB started with pid=56, OS id=13953
Starting background process RBAL
RBAL started with pid=60, OS id=13957
Mon Jun  9 09:53:06 2014
SUCCESS: diskgroup DG was mounted
Mon Jun  9 09:53:10 2014
Setting recovery target incarnation to 2
Mon Jun  9 09:53:11 2014
Successful mount of redo thread 2, with mount id 4128760447
Mon Jun  9 09:53:11 2014
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Mon Jun  9 09:53:11 2014
ALTER DATABASE OPEN
This instance was first to open
Mon Jun  9 09:53:12 2014
Beginning crash recovery of 2 threads
 parallel recovery started with 16 processes
Mon Jun  9 09:53:14 2014
Started redo scan
Mon Jun  9 09:53:14 2014
Completed redo scan
 26 redo blocks read, 8 data blocks need recovery
Mon Jun  9 09:53:14 2014
Started redo application at
 Thread 1: logseq 16276, block 199337, scn 13914083540861
 Thread 2: logseq 13301, block 55078
Mon Jun  9 09:53:14 2014
Recovery of Online Redo Log: Thread 1 Group 2 Seq 16276 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_2.275.803844993
Mon Jun  9 09:53:14 2014
Recovery of Online Redo Log: Thread 2 Group 3 Seq 13301 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Mon Jun  9 09:53:14 2014
Completed redo application
Mon Jun  9 09:53:14 2014
Completed crash recovery at
 Thread 1: logseq 16276, block 199337, scn 13914083560862
 Thread 2: logseq 13301, block 55104, scn 13914083661134
 8 data blocks read, 8 data blocks written, 26 redo blocks read
Attempt to switch log for thread 1 failed. reason: 3
Picked broadcast on commit scheme to generate SCNs
Mon Jun  9 09:53:14 2014
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=58, OS id=14055
Mon Jun  9 09:53:14 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=35, OS id=14062
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Jun  9 09:53:14 2014
ARC0: Becoming the heartbeat ARCH
Mon Jun  9 09:53:17 2014
Thread 2 advanced to log sequence 13302 (thread open)
Thread 2 opened at log sequence 13302
  Current log# 4 seq# 13302 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Successful open of redo thread 2
Mon Jun  9 09:53:17 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Jun  9 09:53:17 2014
SMON: enabling cache recovery
Mon Jun  9 09:53:18 2014
ARC0: Switch failed
Mon Jun  9 09:53:18 2014
Successfully onlined Undo Tablespace 35.
Mon Jun  9 09:53:18 2014
SMON: enabling tx recovery
Mon Jun  9 09:53:18 2014
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 4, CPUs = 20
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=62, OS id=14171
Mon Jun  9 09:53:19 2014
Thread 1 advanced to log sequence 16277 (archiver wakeup)
Mon Jun  9 09:53:22 2014
Completed: ALTER DATABASE OPEN
Mon Jun  9 09:53:26 2014
ALTER SYSTEM SET service_names='rac' SCOPE=MEMORY SID='dqb2';
Mon Jun  9 10:01:02 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_j001_14197.trc:
ORA-00600: internal error code, arguments: [6749], [3], [12662723], [109], [], [], [], []
Mon Jun  9 10:01:06 2014
Trace dumping is performing id=[cdmp_20140609100106]
Mon Jun  9 10:03:24 2014
ALTER SYSTEM SET service_names='rac','dqb' SCOPE=MEMORY SID='dqb2';

叫同事手工启动ASM实例。一步步看报错信息

export ORACLE_SID=+ASM1

sqlplus / as sysdba

sql>startup;

提示ora-29701。

客户报oracle数据库出现了ora-29701错误，查看oracle的 error message

ORA-29701: unable to connect to Cluster Manager

Cause: Connect to CM failed or timed out.

Action: Verify that the CM was started. If the CM was not started, start it and then retry the database startup.

           If the CM died or is not responding, check the Oracle and CM trace files for errors.

       当时以为肯定是cluster出问题了，可是客户说查询都能够运行，就是运行create table语句就出现这个错误，怪了！ 后来通过查看数  据库的告警日志的过程中发现原来是$ORACLE_HOME的空间满了，通过删除没有的trace文件。数据库一切正常！

汗， 后来让同事尝试使用手工启动ASM实例，提示Ora-29701，汗，原来是磁盘空间满