ORA07445 [SIGBUS] [Object specific hardware error]错误一例

一套Solaris上的9.2.0.7系统，实例意外终止，告警日志中出现以下记录:

Thu Sep 2 02:15:41 2010
Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Thu Sep 2 02:15:48 2010
Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_pmon_11379.trc:
ORA-00474: SMON process terminated with error
Thu Sep 2 02:15:48 2010
PMON: terminating instance due to error 474
Wed Sep 1 15:04:20 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_1772.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA78000] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5316.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C843DC8] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5207.trc:
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C887A70] [] []
Wed Sep 1 15:06:24 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_28532.trc:
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C9B7FE8] [] []
Wed Sep 1 15:06:31 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_4392.trc:
ORA-07445: exception encountered: core dump [0000000100624600] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA80000] [] []
Wed Sep 1 15:06:33 2010
Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_3748.trc:
ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error]

trace日志:

Dump file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System name: SunOS

2010-09-02 02:15:41.917
*** SESSION ID:(7.1) 2010-09-02 02:15:33.090
Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0xffffffff7cb3bf90, PC: [0x101e05500, 0000000101E05500]
*** 2010-09-02 02:15:41.936
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] []
Current SQL statement for this session:
update sys.col_usage$ set equality_preds = equality_preds + decode(bitand(:flag,1),0,0,1), equijoin_preds = equijoin_preds + decode(bitand(:flag,2),0,0,1), nonequijoin_preds = nonequijoin_preds + decode(bitand(:flag,4),0,0,1), range_preds = range_preds + decode(bitand(:flag,8),0,0,1), like_preds = like_preds + decode(bitand(:flag,16),0,0,1), null_preds = null_preds + decode(bitand(:flag,32),0,0,1), timestamp = :time where obj# = :objn and intcol# = :coln
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFE3EE8 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFEAF80 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghbshrt()+96 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFEAF80 ?
FFFFFFFF7FFEACA0 ?
000000000 ? 00000BF78 ?
0B38F0000 ?
kghalo()+612 CALL kghbshrt() 103788EC8 ? 000000000 ?
10378C2E0 ?
FFFFFFFF7CB30018 ?
000000458 ? 080000000 ?
kghgex()+336 CALL kghalo() 000000000 ?
FFFFFFFF7CB30018 ?
103788EC8 ? 000001000 ?
FFFFFFFF7CB3E120 ?
C0B38F000000C3D1 ?
kghalf()+316 CALL kghgex() 000007C00 ? 000000440 ?
000000228 ? 103788EC8 ?
000007FFF ? 000000000 ?
kdb4chk()+216 PTR_CALL 0000000000000000 103788EC8 ? 000000178 ?
FFFFFFFF7CB3E180 ?
FFFFFFFF7CB3E150 ?
000000000 ? 102DA35D0 ?
kd4chk()+88 CALL kdb4chk() 3F5484014 ? 1007E7780 ?
000000000 ? 000000000 ?
1007E76C0 ? 000000000 ?
kcbchk()+244 PTR_CALL 0000000000000000 3F5484000 ? 000000000 ?
000002000 ? 000000000 ?
1007E76C0 ? 1007E7720 ?
kcoapl()+1288 CALL kcbchk() 3F5484000 ? 0001007E7 ?
1007E7000 ? 101BA4440 ?
000100400 ? 0001007E7 ?
kcbapl()+96 CALL kcoapl() 000000002 ? 000000000 ?
000000001 ? 3F5484000 ?
10388DA08 ? 3F5486000 ?
kcrfwr()+1716 CALL kcbapl() 10388D990 ? 3F5FA6200 ?
FFFFFFFF7FFEC3B0 ?
00010378E ? 000000004 ?
000000004 ?
kcbchg1()+2056 CALL kcrfwr() FFFFFFFF7FFEC638 ?
40E36C218 ? 10378F5E8 ?
FFFFFFFF7FFEC3B0 ?
10378F690 ? 000000002 ?
ktuchg()+1044 CALL kcbchg1() 000000000 ?
FFFFFFFF7FFECE08 ?
000000000 ? 10378B530 ?
000000000 ?
FFFFFFFF7FFEC730 ?
ktbchg2()+132

Dump file /u01/app/oracle/admin/preg063/udump/preg063_ora_10110.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920preg063
System name: SunOS


2010-09-01 00:29:31.655
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CAAC000] [] []
Current SQL statement for this session:
select lxOid,lxVal from lxString_c83b5454 where lxOid in (:va,:vb,:vc,:vd,:ve,:vf,:vg,:vh,:vi,:vj,:vk,:vl,:vm,:vn,:vo,:vp,:vq,:vr,:vs,:vt,:vu,:vv,:vw,:vx,:vy,:vz,:v26,:v27,:v28,:v29,:v30,:v31,:v32,:v33,:v34,:v35,:v36,:v37,:v38,:v39,:v40,:v41,:v42,:v43,:v44,:v45,:v46,:v47,:v48,:v49,:v50,:v51,:v52,:v53,:v54,:v55,:v56,:v57,:v58,:v59) and lxType=:v60
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ?
000000000 ? 103357D68 ?
00000003E ?
FFFFFFFF7FFF3B98 ?
ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ?
103788C68 ? 10378D000 ?
000102C00 ? 000000000 ?
sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ?
FFFFFFFF7FFFAC30 ?
103790000 ? 10378DE30 ?
000000000 ? 103790DB0 ?
kghalf()+436 PTR_CALL 0000000000000000 00000000A ?
FFFFFFFF7FFFAC30 ?
FFFFFFFF7FFFA950 ?
103788EC8 ? 0B38F0000 ?
B38F0000000000 ?
qerhjManageListElem CALL kghalf() 103788EC8 ?
ent()+116 10B38F0000000001 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CC73418 ?
FFFFFFFF7CAABFF8 ?
1034CC568 ?
qerhjadf()+248 CALL qerhjManageListElem FFFFFFFF7CBA2568 ?
ent() FFFFFFFF7CC733A8 ?
000000002 ? 000000049 ?
FFFFFFFF7CA41E9A ?
0000000AF ?
qerhjBuildHashTable CALL qerhjadf() FFFFFFFF7FFFB0F8 ?
()+4236 FFFFFFFF7C9D2DB0 ?
000000013 ?
FFFFFFFF7C9D2D90 ?
FFFFFFFF7CA41E9A ?
000000001 ?
qerhjFetch()+772 CALL qerhjBuildHashTable 000000000 ? 1038698D8 ?
() 000000040 ?
FFFFFFFF7C9B16A8 ?
000000001 ?
FFFFFFFF7CBA2630 ?
qervwFetch()+148 PTR_CALL 0000000000000000 43C80CBB0 ? 101624E80 ?
FFFFFFFF7FFFB338 ?
00000001E ? 000000008 ?
00000005F ?
kpofrws()+124 PTR_CALL 0000000000000000 43C80CB58 ? 100FEF620 ?
FFFFFFFF7FFFB940 ?
00000001E ? 000101400 ?
000101624 ?
opifch2()+1704 CALL kpofrws() 000000004 ? 000000004 ?
100FEF620 ?
FFFFFFFF7FFFB940 ?
102EC9160 ? 000000010 ?
opiall0()+3900 CALL opifch2() 10101B000 ? 000000006 ?

提交SR后，MOS认为可能是swap空间不足导致，建议检查系统日志:"Please check your OS logs for any errors during this time.Also please check swap space. This error is sometimes recorded when swap space becomes full"。 系统日志显示在Sep 2出现过/tmp文件系统空间耗尽，swap空间不足的问题:

Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Aug 15 02:48:06 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded

该内部错误通过增大swap最后解决了，that's great!