One oracle database crushed due to NFS issue. The database binary was mounted from another server A. So once the NFS was disconnected, the database was crushed. I tried to start it up today but hit below issue.
SQL> startup ORACLE instance started. Total System Global Area 2138521600 bytes Fixed Size 2161024 bytes Variable Size 1761609344 bytes Database Buffers 369098752 bytes Redo Buffers 5652480 bytes ORA-01102: cannot mount database in EXCLUSIVE mode
Then I try to check the alert log. But alert log for oracle 11g is stored in a different location. So I check the parameter background_dump_dest to find the alert log.
SQL> show parameter background
NAME TYPE VALUE
------------------------------------ --------------------------------- ------------------------------
background_core_dump string partial
background_dump_dest string /opt/oracle/diag/rdbms/mrad/mr
ad/trace
The alert log shows below error
ORACLE_BASE from environment = /opt/oracle Tue May 28 16:28:44 2013 ALTER DATABASE MOUNT sculkget: failed to lock /opt/oracle/product/11.2/mrad/dbs/lkMRAD exclusive sculkget: lock held by PID: 1078 ORA-09968: unable to lock file SVR4 Error: 11: Resource temporarily unavailable Additional information: 1078 ORA-1102 signalled during: ALTER DATABASE MOUNT... Tue May 28 16:35:08 2013 alter database mount
The file /opt/oracle/product/11.2/mrad/dbs/lkMRAD is an lock file so I renamed it. And try to restart again. Now the error is
SQL> alter database mount; alter database mount * ERROR at line 1: ORA-00205: error in identifying control file, check alert log for more info
Check the alert log find below
ALTER DATABASE MOUNT ORA-00210: cannot open the specified control file ORA-00202: control file: '/opt/oracle/oradata/mrad/sys03/control03.ctl' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 1084 ORA-00210: cannot open the specified control file ORA-00202: control file: '/opt/oracle/oradata/mrad/sys02/control02.ctl' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 1084 ORA-00210: cannot open the specified control file ORA-00202: control file: '/opt/oracle/oradata/mrad/sys01/control01.ctl' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 1084
It says there is lock on control file. This is because the NFS lock was not released properly due to last time crush. So I run below command to release the lock.
-bash-4.1# clear_locks -s A Clearing locks held for NFS client B on server A clear of locks held for B on A returned success
Then start the database. Everything is ok