Aix 7.1
参考文档：
https://blogs.oracle.com/database4cn/rac
Resolving ORA-481 and "terminating the instance due to error 481" (Doc ID 1950963.1)
ORA-00481 After "The instance eviction reason is 0x2" due to Lack of Ticket (Doc ID 1644015.1)
归根结底是RAC节点之间消息传输流量控制的问题，除网络、硬件原因外，需要打补丁解决。
以下为第一次处理前具体的alertlog。
◆node1 Wed May 24 01:59:24 2017 Remote instance 2 kill is issued with system inc 228 LMON received an instance eviction notification from instance 1 The instance eviction reason is 0x2 The instance eviction map is 2 Reconfiguration started (old inc 228, new inc 230) List of instances: 1 (myinst: 1) Wed May 24 01:59:27 2017 Trace dumping is performing id=[cdmp_20170524015904] Global Resource Directory frozen * dead instance detected - domain 0 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 24 01:59:28 2017 LMS 1: 14 GCS shadows cancelled, 1 closed, 0 Xw survived Wed May 24 01:59:28 2017 LMS 0: 23 GCS shadows cancelled, 0 closed, 0 Xw survived Wed May 24 01:59:28 2017 LMS 2: 21 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Wed May 24 01:59:32 2017 Instance recovery: looking for dead threads Beginning instance recovery of 1 threads Wed May 24 01:59:48 2017 parallel recovery started with 32 processes Started redo scan Wed May 24 01:59:49 2017 Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed May 24 02:00:00 2017 Completed redo scan read 2094379 KB redo, 237345 data blocks need recovery Wed May 24 02:00:03 2017 Reconfiguration started (old inc 230, new inc 232) List of instances: 1 2 (myinst: 1) Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 24 02:00:04 2017 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Wed May 24 02:00:04 2017 Wed May 24 02:00:04 2017 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Wed May 24 02:02:27 2017 ， ，此处大量12170 ， Wed May 24 02:05:34 2017 LMON (ospid: 35979900): terminating the instance due to error 481 Wed May 24 02:05:34 2017 System state dump is made for local instance System State dumped to trace file /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_40894656.trc Instance terminated by LMON, pid = 35979900 Wed May 24 08:12:27 2017 Starting ORACLE instance (normal) sskgpgetexecname failed to get name LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 en1 192.168.0.0 configured from GPnP Profile for use as a cluster interconnect Interface type 1 en0 10.209.199.0 configured from GPnP Profile for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options. Using parameter settings in server-side pfile /u01/app/11.2.0/oracle/product/11.2.0/db_1/dbs/initorcl1.ora System parameters with non-default values: processes = 2000 timed_statistics = TRUE sga_max_size = 70G spfile = "+DATA1/orcl/spfileorcl.ora" sga_target = 50G control_files = "+DATA1/orcl/controlfile/current.261.832429113" control_files = "+ARCDG/orcl/controlfile/current.259.832429113" db_block_size = 8192 compatible = "11.2.0.0.0" log_archive_format = "%t_%s_%r.dbf" cluster_database = TRUE db_create_file_dest = "+DATA1" db_recovery_file_dest = "+ARCDG" db_recovery_file_dest_size= 2048G thread = 1 undo_tablespace = "UNDOTBS1" undo_retention = 10800 instance_number = 1 remote_login_passwordfile= "NONE" db_domain = "" service_names = "orcl" dispatchers = "(PROTOCOL=TCP) (SERVICE=orclXDB)" local_listener = "(ADDRESS = (PROTOCOL = TCP)(HOST = 10.209.199.4)(PORT = 1521))" remote_listener = "rac-scan:1521" result_cache_max_size = 268736K audit_file_dest = "/u01/app/11.2.0/oracle/admin/orcl/adump" audit_trail = "DB" db_name = "orcl" open_cursors = 1000 sql_trace = FALSE optimizer_index_caching = 90 pga_aggregate_target = 20G deferred_segment_creation= FALSE aq_tm_processes = 5 diagnostic_dest = "/u01/app/11.2.0/oracle" Deprecated system parameters with specified values: sql_trace End of deprecated system parameter listing Cluster communication is configured to use the following interface(s) for this instance 192.168.0.1 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed May 24 08:12:32 2017 PMON started with pid=2, OS id=35652038 Wed May 24 08:12:32 2017 VKTM started with pid=3, OS id=33489120 at elevated priority VKTM running at (10)millisec precision with DBRM quantum (100)ms Wed May 24 08:12:32 2017 GEN0 started with pid=4, OS id=34209898 Wed May 24 08:12:32 2017 DIAG started with pid=5, OS id=36438538 Wed May 24 08:12:33 2017 DBRM started with pid=6, OS id=24576446 Wed May 24 08:12:33 2017 PING started with pid=7, OS id=29687920 Wed May 24 08:12:33 2017 PSP0 started with pid=8, OS id=36635296 Wed May 24 08:12:33 2017 ACMS started with pid=9, OS id=34341462 Wed May 24 08:12:33 2017 DIA0 started with pid=10, OS id=32047870 Wed May 24 08:12:33 2017 LMON started with pid=11, OS id=36045304 Wed May 24 08:12:35 2017 LMD0 started with pid=12, OS id=31391860 Wed May 24 08:12:35 2017 LMS0 started with pid=13, OS id=35979918 at elevated priority Wed May 24 08:12:35 2017 LMS1 started with pid=14, OS id=22741294 at elevated priority Wed May 24 08:12:36 2017 LMS2 started with pid=15, OS id=29229354 at elevated priority Wed May 24 08:12:36 2017 RMS0 started with pid=16, OS id=20185780 Wed May 24 08:12:36 2017 LMHB started with pid=17, OS id=29425998 Wed May 24 08:12:36 2017 MMAN started with pid=18, OS id=9699954 Wed May 24 08:12:36 2017 DBW0 started with pid=19, OS id=31850686 Wed May 24 08:12:36 2017 DBW1 started with pid=20, OS id=36045442 Wed May 24 08:12:36 2017 DBW2 started with pid=21, OS id=40894536 Wed May 24 08:12:36 2017 DBW3 started with pid=22, OS id=29819564 Wed May 24 08:12:36 2017 DBW4 started with pid=23, OS id=36634890 Wed May 24 08:12:36 2017 LGWR started with pid=24, OS id=35783370 Wed May 24 08:12:37 2017 CKPT started with pid=25, OS id=35914258 Wed May 24 08:12:37 2017 SMON started with pid=26, OS id=32374796 Wed May 24 08:12:37 2017 RECO started with pid=27, OS id=35652280 Wed May 24 08:12:37 2017 RBAL started with pid=28, OS id=36438322 Wed May 24 08:12:37 2017 ASMB started with pid=29, OS id=39387144 Wed May 24 08:12:37 2017 MMON started with pid=30, OS id=2621900 NOTE: initiating MARK startup Wed May 24 08:12:37 2017 Starting background process MARKMMNL started with pid=31, OS id=35979732 starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'... Wed May 24 08:12:37 2017 MARK started with pid=32, OS id=35717688 NOTE: MARK has subscribed starting up 1 shared server(s) ... lmon registered with NM - instance number 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 236) List of instances: 1 2 (myinst: 1) Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid according to instance 2 * domain 0 valid = 1 according to instance 2 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Wed May 24 08:12:45 2017 Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed May 24 08:12:52 2017 LCK0 started with pid=34, OS id=24379778 Wed May 24 08:12:52 2017 Starting background process RSMN Wed May 24 08:12:52 2017 RSMN started with pid=36, OS id=30802026 ORACLE_BASE from environment = /u01/app/11.2.0/oracle Wed May 24 08:12:53 2017 ALTER DATABASE MOUNT Wed May 24 08:12:53 2017 NOTE: Loaded library: System Wed May 24 08:12:53 2017 SUCCESS: diskgroup DATA1 was mounted SUCCESS: diskgroup ARCDG was mounted Wed May 24 08:12:53 2017 NOTE: dependency between database orcl and diskgroup resource ora.DATA1.dg is established NOTE: dependency between database orcl and diskgroup resource ora.ARCDG.dg is established Wed May 24 08:12:57 2017 Successful mount of redo thread 1, with mount id 1472569957 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Lost write protection disabled Completed: ALTER DATABASE MOUNT Wed May 24 08:12:58 2017 ALTER DATABASE OPEN Block change tracking file is current. Picked broadcast on commit scheme to generate SCNs Wed May 24 08:12:58 2017 SUCCESS: diskgroup DATA2 was mounted NOTE: dependency between database orcl and diskgroup resource ora.DATA2.dg is established SUCCESS: diskgroup DATA3 was mounted NOTE: dependency between database orcl and diskgroup resource ora.DATA3.dg is established Thread 1 advanced to log sequence 255944 (thread open) Thread 1 opened at log sequence 255944 Current log# 2 seq# 255944 mem# 0: +ARCDG/orcl/onlinelog/group_2.261.840376035 Current log# 2 seq# 255944 mem# 1: +ARCDG/orcl/onlinelog/group_2.2113.840376039 Current log# 2 seq# 255944 mem# 2: +ARCDG/orcl/onlinelog/group_2.2114.840376041 Current log# 2 seq# 255944 mem# 3: +ARCDG/orcl/onlinelog/group_2.2116.840376043 Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Starting background process CTWR Wed May 24 08:12:59 2017 CTWR started with pid=39, OS id=32768642 Block change tracking service is active. Wed May 24 08:13:00 2017 SMON: enabling cache recovery Successfully onlined Undo Tablespace 2. Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery Database Characterset is AL32UTF8 No Resource Manager plan active Starting background process GTX0 Wed May 24 08:13:07 2017 GTX0 started with pid=42, OS id=35062526 Starting background process RCBG Wed May 24 08:13:07 2017 RCBG started with pid=43, OS id=27721944 replication_dependency_tracking turned off (no async multimaster replication found) Wed May 24 08:13:08 2017 Starting background process QMNC Wed May 24 08:13:08 2017 QMNC started with pid=44, OS id=34013894 Completed: ALTER DATABASE OPEN Wed May 24 08:13:17 2017 Starting background process CJQ0 Wed May 24 08:13:17 2017 CJQ0 started with pid=41, OS id=11469004 Wed May 24 08:13:43 2017 Starting background process SMCO Wed May 24 08:13:43 2017 SMCO started with pid=120, OS id=42336480 ， ， ， Wed May 24 09:08:16 2017 Errors in file /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_j001_33161436.trc: ORA-12012: 鑎仟錨狮鎊骚琛^啼絕茕竈239 錨呛閊証 ORA-00001: 杩^蒎^蟐湾^辕涓^犁害鎊較哝^荨浠(CM.UK_LTE_ZY_CHECK_HISTORY) ORA-06512: 錨塄 "CM.JOB_LTE_ZY_CHECK", line 27 ORA-06512: 錨塄 line 1 ◆node2 Wed May 24 01:59:01 2017 LMS1 (ospid: 26411086) received an instance eviction notification from instance 1 [2] Wed May 24 01:59:01 2017 LMON received an instance eviction notification from instance 1 The instance eviction reason is 0x2 The instance eviction map is 2 Wed May 24 01:59:04 2017 PMON (ospid: 50725354): terminating the instance due to error 481 Wed May 24 01:59:04 2017 System state dump is made for local instance System State dumped to trace file /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_diag_42336686.trc Wed May 24 01:59:04 2017 ORA-1092 : opitsk aborting process Wed May 24 01:59:04 2017 License high water mark = 516 Instance terminated by PMON, pid = 50725354 USER (ospid: 50987430): terminating the instance Instance terminated by USER, pid = 50987430 Wed May 24 01:59:29 2017 Starting ORACLE instance (normal) sskgpgetexecname failed to get name LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 en1 192.168.0.0 configured from GPnP Profile for use as a cluster interconnect Interface type 1 en0 10.209.199.0 configured from GPnP Profile for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options. Using parameter settings in server-side pfile /u01/app/11.2.0/oracle/product/11.2.0/db_1/dbs/initorcl2.ora System parameters with non-default values: processes = 2000 timed_statistics = TRUE sga_max_size = 70G spfile = "+DATA1/orcl/spfileorcl.ora" sga_target = 50G control_files = "+DATA1/orcl/controlfile/current.261.832429113" control_files = "+ARCDG/orcl/controlfile/current.259.832429113" db_block_size = 8192 compatible = "11.2.0.0.0" log_archive_format = "%t_%s_%r.dbf" cluster_database = TRUE db_create_file_dest = "+DATA1" db_recovery_file_dest = "+ARCDG" db_recovery_file_dest_size= 2048G thread = 2 undo_tablespace = "UNDOTBS2" undo_retention = 10800 instance_number = 2 remote_login_passwordfile= "NONE" db_domain = "" dispatchers = "(PROTOCOL=TCP) (SERVICE=orclXDB)" local_listener = "(ADDRESS = (PROTOCOL = TCP)(HOST = 10.209.199.5)(PORT = 1521))" remote_listener = "rac-scan:1521" result_cache_max_size = 268736K audit_file_dest = "/u01/app/11.2.0/oracle/admin/orcl/adump" audit_trail = "DB" db_name = "orcl" open_cursors = 1000 sql_trace = FALSE optimizer_index_caching = 90 pga_aggregate_target = 20G deferred_segment_creation= FALSE aq_tm_processes = 5 diagnostic_dest = "/u01/app/11.2.0/oracle" Deprecated system parameters with specified values: sql_trace End of deprecated system parameter listing Cluster communication is configured to use the following interface(s) for this instance 192.168.0.2 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed May 24 01:59:34 2017 PMON started with pid=2, OS id=66584656 Wed May 24 01:59:34 2017 VKTM started with pid=3, OS id=66846744 at elevated priority VKTM running at (10)millisec precision with DBRM quantum (100)ms Wed May 24 01:59:34 2017 GEN0 started with pid=4, OS id=26608090 Wed May 24 01:59:34 2017 DIAG started with pid=5, OS id=26083716 Wed May 24 01:59:34 2017 DBRM started with pid=6, OS id=24510972 Wed May 24 01:59:34 2017 PING started with pid=7, OS id=65077306 Wed May 24 01:59:34 2017 PSP0 started with pid=8, OS id=66781402 Wed May 24 01:59:34 2017 ACMS started with pid=9, OS id=66978040 Wed May 24 01:59:34 2017 DIA0 started with pid=10, OS id=66519050 Wed May 24 01:59:34 2017 LMON started with pid=11, OS id=66453694 Wed May 24 01:59:37 2017 LMD0 started with pid=12, OS id=23658758 Wed May 24 01:59:37 2017 LMS0 started with pid=13, OS id=66322614 at elevated priority Wed May 24 01:59:37 2017 LMS1 started with pid=14, OS id=65798210 at elevated priority Wed May 24 01:59:37 2017 LMS2 started with pid=15, OS id=15204812 at elevated priority Wed May 24 01:59:38 2017 RMS0 started with pid=16, OS id=65732820 Wed May 24 01:59:38 2017 LMHB started with pid=17, OS id=65339438 Wed May 24 01:59:38 2017 MMAN started with pid=18, OS id=52298036 Wed May 24 01:59:38 2017 DBW0 started with pid=19, OS id=52232458 Wed May 24 01:59:38 2017 DBW1 started with pid=20, OS id=65273872 Wed May 24 01:59:38 2017 DBW2 started with pid=21, OS id=65208354 Wed May 24 01:59:38 2017 DBW3 started with pid=22, OS id=65994980 Wed May 24 01:59:38 2017 DBW4 started with pid=23, OS id=61210752 Wed May 24 01:59:38 2017 LGWR started with pid=24, OS id=6095298 Wed May 24 01:59:38 2017 CKPT started with pid=25, OS id=65470648 Wed May 24 01:59:38 2017 SMON started with pid=26, OS id=64946374 Wed May 24 01:59:38 2017 RECO started with pid=27, OS id=51642854 Wed May 24 01:59:39 2017 RBAL started with pid=28, OS id=64618728 Wed May 24 01:59:39 2017 ASMB started with pid=29, OS id=64553208 Wed May 24 01:59:39 2017 MMON started with pid=30, OS id=64422054 NOTE: initiating MARK startup Wed May 24 01:59:39 2017 MMNL started with pid=31, OS id=42336716 Starting background process MARK starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'... Wed May 24 01:59:39 2017 MARK started with pid=32, OS id=51577168 NOTE: MARK has subscribed Wed May 24 01:59:39 2017 starting up 1 shared server(s) ... lmon registered with NM - instance number 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 232) List of instances: 1 2 (myinst: 2) Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 0 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Wed May 24 01:59:48 2017 Submitted all GCS remote-cache requests Wed May 24 02:05:10 2017 Trace dumping is performing id=[cdmp_20170524020534] Wed May 24 02:05:18 2017 Reconfiguration started (old inc 232, new inc 234) List of instances: 2 (myinst: 2) Nested reconfiguration detected. Global Resource Directory frozen * dead instance detected - domain 0 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 24 02:05:19 2017 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Wed May 24 02:05:19 2017 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Wed May 24 02:05:19 2017 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed May 24 02:05:25 2017 LCK0 started with pid=34, OS id=51118396 Wed May 24 02:05:25 2017 Starting background process RSMN Wed May 24 02:05:25 2017 RSMN started with pid=36, OS id=51773786 ORACLE_BASE not set in environment. It is recommended that ORACLE_BASE be set in the environment Reusing ORACLE_BASE from an earlier startup = /u01/app/11.2.0/oracle Wed May 24 02:05:25 2017 ALTER DATABASE MOUNT This instance was first to mount Wed May 24 02:05:26 2017 NOTE: Loaded library: System Wed May 24 02:05:26 2017 SUCCESS: diskgroup DATA1 was mounted Wed May 24 02:05:26 2017 NOTE: dependency between database orcl and diskgroup resource ora.DATA1.dg is established SUCCESS: diskgroup ARCDG was mounted NOTE: dependency between database orcl and diskgroup resource ora.ARCDG.dg is established Wed May 24 02:05:30 2017 Successful mount of redo thread 2, with mount id 1472569957 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Lost write protection disabled Completed: ALTER DATABASE MOUNT ALTER DATABASE OPEN This instance was first to open Wed May 24 02:05:30 2017 SUCCESS: diskgroup DATA2 was mounted NOTE: dependency between database orcl and diskgroup resource ora.DATA2.dg is established SUCCESS: diskgroup DATA3 was mounted NOTE: dependency between database orcl and diskgroup resource ora.DATA3.dg is established Block change tracking file is current. Beginning crash recovery of 2 threads parallel recovery started with 32 processes Started redo scan Wed May 24 02:05:44 2017 Completed redo scan read 2158203 KB redo, 230150 data blocks need recovery Started redo application at Thread 1: logseq 255942, block 163388 Thread 2: logseq 236286, block 2067920 Recovery of Online Redo Log: Thread 1 Group 3 Seq 255942 Reading mem 0 Mem# 0: +ARCDG/orcl/onlinelog/group_3.2085.840375427 Mem# 1: +ARCDG/orcl/onlinelog/group_3.2084.840375445 Mem# 2: +ARCDG/orcl/onlinelog/group_3.263.840375447 Mem# 3: +ARCDG/orcl/onlinelog/group_3.2089.840375449 Recovery of Online Redo Log: Thread 2 Group 7 Seq 236286 Reading mem 0 Mem# 0: +ARCDG/orcl/onlinelog/group_7.2049.840374587 Mem# 1: +ARCDG/orcl/onlinelog/group_7.2052.840374591 Mem# 2: +ARCDG/orcl/onlinelog/group_7.2057.840374593 Mem# 3: +ARCDG/orcl/onlinelog/group_7.2058.840374597 Recovery of Online Redo Log: Thread 2 Group 5 Seq 236287 Reading mem 0 Mem# 0: +ARCDG/orcl/onlinelog/group_5.2025.840374445 Mem# 1: +ARCDG/orcl/onlinelog/group_5.2026.840374483 Mem# 2: +ARCDG/orcl/onlinelog/group_5.2029.840374489 Mem# 3: +ARCDG/orcl/onlinelog/group_5.2034.840374493 Wed May 24 02:06:05 2017 Completed redo application of 1702.07MB Completed crash recovery at Thread 1: logseq 255942, block 291036, scn 12495715727030 Thread 2: logseq 236287, block 2742038, scn 12495715600520 230150 data blocks read, 229989 data blocks written, 2158203 redo k-bytes read Thread 1 advanced to log sequence 255943 (thread recovery) Picked broadcast on commit scheme to generate SCNs Wed May 24 02:06:06 2017 Thread 2 advanced to log sequence 236288 (thread open) Thread 2 opened at log sequence 236288 Current log# 6 seq# 236288 mem# 0: +ARCDG/orcl/onlinelog/group_6.2040.840374575 Current log# 6 seq# 236288 mem# 1: +ARCDG/orcl/onlinelog/group_6.2041.840374579 Current log# 6 seq# 236288 mem# 2: +ARCDG/orcl/onlinelog/group_6.2042.840374581 Current log# 6 seq# 236288 mem# 3: +ARCDG/orcl/onlinelog/group_6.2048.840374585 Successful open of redo thread 2 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Starting background process CTWR Wed May 24 02:06:07 2017 CTWR started with pid=89, OS id=49349114 Block change tracking service is active. Wed May 24 02:06:07 2017 SMON: enabling cache recovery Successfully onlined Undo Tablespace 4. Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery Database Characterset is AL32UTF8 No Resource Manager plan active Starting background process GTX0 Wed May 24 02:06:11 2017 GTX0 started with pid=98, OS id=42992042 Starting background process RCBG Wed May 24 02:06:11 2017 RCBG started with pid=52, OS id=40370648 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC Wed May 24 02:06:13 2017 QMNC started with pid=119, OS id=47514086 Completed: ALTER DATABASE OPEN SMON: Parallel transaction recovery tried Starting background process SMCO Wed May 24 02:06:17 2017 SMCO started with pid=153, OS id=50724962 Wed May 24 02:06:17 2017 db_recovery_file_dest_size of 2097152 MB is 4.35% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. Wed May 24 02:06:19 2017 Starting background process CJQ0 Wed May 24 02:06:19 2017 CJQ0 started with pid=46, OS id=43581760 Wed May 24 02:07:35 2017 Thread 2 advanced to log sequence 236289 (LGWR switch) Current log# 7 seq# 236289 mem# 0: +ARCDG/orcl/onlinelog/group_7.2049.840374587 Current log# 7 seq# 236289 mem# 1: +ARCDG/orcl/onlinelog/group_7.2052.840374591 Current log# 7 seq# 236289 mem# 2: +ARCDG/orcl/onlinelog/group_7.2057.840374593 Current log# 7 seq# 236289 mem# 3: +ARCDG/orcl/onlinelog/group_7.2058.840374597 Wed May 24 02:08:09 2017 Thread 2 advanced to log sequence 236290 (LGWR switch) Current log# 5 seq# 236290 mem# 0: +ARCDG/orcl/onlinelog/group_5.2025.840374445 Current log# 5 seq# 236290 mem# 1: +ARCDG/orcl/onlinelog/group_5.2026.840374483 Current log# 5 seq# 236290 mem# 2: +ARCDG/orcl/onlinelog/group_5.2029.840374489 Current log# 5 seq# 236290 mem# 3: +ARCDG/orcl/onlinelog/group_5.2034.840374493 Wed May 24 02:08:51 2017 Thread 2 advanced to log sequence 236291 (LGWR switch) Current log# 6 seq# 236291 mem# 0: +ARCDG/orcl/onlinelog/group_6.2040.840374575 Current log# 6 seq# 236291 mem# 1: +ARCDG/orcl/onlinelog/group_6.2041.840374579 Current log# 6 seq# 236291 mem# 2: +ARCDG/orcl/onlinelog/group_6.2042.840374581 Current log# 6 seq# 236291 mem# 3: +ARCDG/orcl/onlinelog/group_6.2048.840374585 Wed May 24 02:09:30 2017 Thread 2 advanced to log sequence 236292 (LGWR switch) Current log# 7 seq# 236292 mem# 0: +ARCDG/orcl/onlinelog/group_7.2049.840374587 Current log# 7 seq# 236292 mem# 1: +ARCDG/orcl/onlinelog/group_7.2052.840374591 Current log# 7 seq# 236292 mem# 2: +ARCDG/orcl/onlinelog/group_7.2057.840374593 Current log# 7 seq# 236292 mem# 3: +ARCDG/orcl/onlinelog/group_7.2058.840374597
针对这个RAC集群，做了加多的配置修正、优化，主要是如下所述。
◆数据库配置优化
0、前期大量的日志切换无法完成，导致数据库挂起，为此新增了loggroup，并删掉无用多余的logmember。
1、增大了ASM实例的内存，由350M--->2G
2、RAC并行进程强制在本地实例分配
3、为减少对性能的影响，关闭audit审计
4、AWR 资料保存时间由7天增加到30天
5、SGA增大10G至60G
◆修改网络相关的系统核心参数
/usr/sbin/no -p -o tcp_sendspace=4194304
/usr/sbin/no -p -o tcp_recvspace=4194304
/usr/sbin/no -p -o rfc1323=1
/usr/sbin/no -p -o sb_max=8388608
/usr/sbin/no -p -o udp_ephemeral_low=9000
/usr/sbin/no -p -o tcp_ephemeral_low=9000
◆修改内联网卡的参数
chdev -l en1 -a tcp_sendspace=1048576 -a rfc1323=1 -a tcp_recvspace=1048576
◆SWAP
物理内存大于 16GB，但 SWAP 空间低于 16 GB，不符合 Oracle 安装的标准建议。增大PS。chps -s 192 hd6
◆时区及时间确认，安全起见，暂不修改。
◆ntp确认，使用GI集群自身的时间同步，确保各个节点的时间同步，目前是几十秒的偏差，不是问题。
◆HA集群配置修改，当初安装集群时部分内容没有按照官方文档实施，本次做了补救，但不全面，可能仍存在未知的风险。
5月25日重启后，经多一段时间的观察，有所改善，但于6月7日9点又发生一次节点2当。
但此次已经不是节点的驱逐，说明上次的修正与配置优化起到了作用，初步断定与网络参数配置修改相关。日志如下，无evict日志。
Wed Jun 07 09:02:57 2017 Errors in file /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_lms0_7930030.trc (incident=1023882): ORA-00600: internal error code, arguments: [kjbrref:pkey], [3881577], [6], [7493495], [0], [], [], [], [], [], [], [] Incident details in: /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl2/incident/incdir_1023882/orcl2_lms0_7930030_i1023882.trc Wed Jun 07 09:02:59 2017 Trace dumping is performing id=[cdmp_20170607090259] Errors in file /u01/app/11.2.0/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_lms0_7930030.trc: ORA-00600: internal error code, arguments: [kjbrref:pkey], [3881577], [6], [7493495], [0], [], [], [], [], [], [], [] LMS0 (ospid: 7930030): terminating the instance due to error 484 Instance terminated by LMS0, pid = 7930030 Wed Jun 07 09:14:01 2017 Starting ORACLE instance (normal)
针对此次的600内部错误，初步结论：
5月25日优化重启后，6月7日又发生过1次节点down的情况，经排查，基本确认是oracle BUG，在某些情况下会发生节点down。目前数据库PSU版本11.2.0.1.0，需要打补丁到最新版本11.2.0.1.6。后续需择机打补丁。