1. RAC监听服务进程关闭失败问题分析处理
1.1 现象
1.2 分析
1）检查监听进程ps aux|grep tnslsnr|grep -v grep，发现用户grid变成daemon用户启动的进程服务
2）检查相关日志抛出权限异常问题
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Solaris Error: 13: Permission denied
3）上面的现象跟MOS(Doc ID 434062.1)相符，参考其解决方案进行修复
原因：
由于/var/tmp/.oracle或/tmp/.oracle目录设置了“ sticky”标志，因此只有创建它们的用户才能删除这些文件。当存在的历史IPC Socket文件的用户名与当前启动监听器的用户权限不同，从而引起监听器无法在启动时将它们删除，导致监听器启动失败。
1.3 解决方案
# 1. 检查/var/tmp/.oracle隐藏目录是否存在daemon用户相关文件
ls -l /var/tmp/.oracle/* |grep 'daemon'
# 2. 确认存在异常用户文件
daemon sLISTENER
# 3. 杀掉异常监听进程
kill -9 <PID>
# 4. 修改相关文件的用户
chown grid:oinstall sLISTENER
# 5. 重启监听服务恢复正常
srvctl start listener -n node1
MOS解决方案
删除 /var/tmp/.oracle (Linux or Solaris) or (on AIX), /tmp/.oracle后再重启启动服务并创建正常的权限文件。
1. Stop all listeners
2. Remove the /var/tmp/.oracle or /tmp/.oracle directory, e.g. run the following from the root account:
rm -rf /var/tmp/.oracle
3. Check the listener(s) configuration contains unique keys specified for each IPC endpoint and each listener
4. Start the listener(s) from the normal Oracle OS user.
1.4 参考文档
[434062.1](Listener Fails To Start With IPC Permission Errors TNS-12546 TNS-12555 TNS-00516 TNS-00525)
One or more of the following items:
Some IPC keys have been reused from older listener(s)
Same listener has previously been started under different OS users
Listeners running under different OS users are using the same IPC keys
Permissions in /var/tmp or /tmp have been changed recently.
原因
Older IPC socket file(s) were left in /var/tmp/.oracle or /tmp/.oracle and they cannot be deleted by the listener upon startup.
Listener IPC endpoints on UNIX platforms are implemented with local UNIX sockets. IPC key names are used to form the socket file names created in /var/tmp/.oracle or /tmp/.oracle. The listener creates the files with the same user ownership as the OS user it has been started from.
The /var/tmp/.oracle or /tmp/.oracle directory has the "sticky" flag set so the files can be removed only by the user that created them. At listener startup, if the listener finds an older socket file with the same name (same IPC key) but with different user ownership as the user it's running then it will fail to delete this socket file due to OS permission restrictions.
SOLUTION
IMPORTANT NOTE FOR RAC INSTALLATIONS: Please be aware that this step should ONLY be taken as a last resort and ONLY when there are no Net connections to the database(s). This procedure could cause active sessions to be terminated. It is NOT recommended to take this action in a RAC environment. see Note 391790.1 for details.
Clear the /var/tmp/.oracle (Linux or Solaris) or (on AIX), /tmp/.oracle directory and start from fresh:
Stop all listeners
Remove the /var/tmp/.oracle or /tmp/.oracle directory, e.g. run the following from the root account:
rm -rf /var/tmp/.oracle
Check the listener(s) configuration contains unique keys specified for each IPC endpoint and each listener
Start the listener(s) from the normal Oracle OS user.
Clearing the /var/tmp/.oracle or /tmp/.oracle directory will force the listeners to recreate the socket files from fresh, each with its own correct ownership.
You need to make sure that you always start the listener(s) from the same OS user(s) and that you do not reuse IPC key endpoints between listeners running under different users, otherwise the problem will resurface.