• ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302


    --节点1 告警日志

    Thu Jun 20 03:02:18 2019
    Thread 1 advanced to log sequence 2092 (LGWR switch)
    Current log# 11 seq# 2092 mem# 0: +DATA/test/onlinelog/redo11_1.log
    Current log# 11 seq# 2092 mem# 1: +DATA/test/onlinelog/redo11_2.log
    Thu Jun 20 03:02:18 2019
    LNS: Standby redo logfile selected for thread 1 sequence 2092 for destination LOG_ARCHIVE_DEST_2
    Thu Jun 20 03:02:20 2019
    Archived Log entry 5703 added for thread 1 sequence 2091 ID 0x9b8db1ed dest 1:
    Thu Jun 20 03:03:53 2019
    skgxpvfynet: mtype: 61 process 316751 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
    Errors in file /u01/app/oracle/diag/rdbms/test/test1/trace/test1_ora_316751.trc (incident=560025):
    ORA-00603: ORACLE server session terminated by fatal error
    ORA-27504: IPC error creating OSD context
    ORA-27300: OS system dependent operation:sendmsg failed with status: 105
    ORA-27301: OS failure message: No buffer space available
    ORA-27302: failure occurred at: sskgxpsnd2
    Incident details in: /u01/app/oracle/diag/rdbms/test/test1/incident/incdir_560025/test1_ora_316751_i560025.trc
    Thu Jun 20 03:03:54 2019
    Dumping diagnostic data in directory=[cdmp_20190620030354], requested by (instance=1, osid=316751), summary=[incident=560025].
    opiodr aborting process unknown ospid (316751) as a result of ORA-603
    Thu Jun 20 03:04:03 2019
    Sweep [inc][560025]: completed
    Sweep [inc2][560025]: completed
    Thu Jun 20 03:06:15 2019
    Thread 1 advanced to log sequence 2093 (LGWR switch)
    Current log# 5 seq# 2093 mem# 0: +DATA/test/onlinelog/redo05_1.log
    Current log# 5 seq# 2093 mem# 1: +DATA/test/onlinelog/redo05_2.log
    Thu Jun 20 03:06:16 2019


    ---trc 文件--trc文件,初步看应该是服务器 网卡mtu值相关问题
    *** 2019-06-20 03:03:53.395
    *** CLIENT ID:() 2019-06-20 03:03:53.395
    *** SERVICE NAME:() 2019-06-20 03:03:53.395
    *** MODULE NAME:() 2019-06-20 03:03:53.395
    *** ACTION NAME:() 2019-06-20 03:03:53.395

    SKGXP:[7f5ab0ef57c0.0]{0}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).
    SKGXP:[7f5ab0ef57c0.1]{0}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.16.188.
    SKGXP:[7f5ab0ef57c0.2]{0}: phase 'send', 0 tries, 100 loops, 32354 ms (last)
    struct ksxpp * ksxppg_ [0xc11fde0, 0x7f5aadaf5588) = 0x7f5aadaf5580
    Dump of memory from 0x00007F5AADAF5580 to 0x00007F5AADAF6AB0


    ---数据库状态
    select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;

    --添加到CRT
    set lines 1000 alter session set nls_date_format='yyyymmdd hh24:mi:ss'; set time on select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;

    09:42:29 SYS@test1(test1)> set lines 1000
    09:43:30 SYS@test1(test1)> alter session set nls_date_format='yyyymmdd hh24:mi:ss';

    Session altered.

    09:43:31 SYS@test1(test1)> set time on
    09:43:31 SYS@test1(test1)> select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;

    INST_ID INSTANCE_NAME STARTUP_TIME
    ---------- ---------------- -----------------
    1 test1 20190509 15:30:08
    2 test2 20190507 21:37:04

    ---节点2 日志
    Thu Jun 20 03:03:54 2019
    Dumping diagnostic data in directory=[cdmp_20190620030354], requested by (instance=1, osid=316751), summary=[incident=560025].
    Thu Jun 20 03:10:10 2019
    Thread 2 advanced to log sequence 912 (LGWR switch)
    Current log# 6 seq# 912 mem# 0: +DATA/test/onlinelog/redo06_1.log
    Current log# 6 seq# 912 mem# 1: +DATA/test/onlinelog/redo06_2.log


    ---参考官方文档

    2041723.1


    CAUSE
    This happens due to less space available for network buffer reservation.

    SOLUTION
    1. On servers with High Physical Memory, the parameter vm.min_free_kbytes should be set in the order of 0.4% of total Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers reducing the probability of a low-buffer-space conditions.

    *** For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1073742 ***

    On NUMA Enabled Systems, the value of vm.min_free_kbytes should be multiplied by the number of NUMA nodes since the value is to be split across all the nodes.


    On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here 'n' is the number of NUMA nodes.

    2. Additionally, the MTU value should be modified as below

    #ifconfig lo mtu 16436

    To make the change persistent over reboot add the following line in the file /etc/sysconfig/network-scripts/ifcfg-lo :

    MTU=16436
    Save the file and restart the network service to load the changes

    #service network restart

    Note : While making the changes in CRS nodes, if network is restarted while CRS is up, it can hung CRS. So cluster services should be stopped prior to the network restart.

    ---野鸡博客
    http://ju.outofmemory.cn/entry/76102

    ---实际操作,两个节点执行

    [root@test1 bin]# ifconfig lo mtu 16436
    [root@test1 bin]#

    [root@test1 bin]# vi /etc/sysconfig/network-scripts/ifcfg-lo
    DEVICE=lo
    IPADDR=127.0.0.1
    NETMASK=255.0.0.0
    NETWORK=127.0.0.0
    # If you're having problems with gated making 127.0.0.0/8 a martian,
    # you can change this to something else (255.255.255.255, for example)
    BROADCAST=127.255.255.255
    ONBOOT=yes
    NAME=loopback
    MTU=16436

    ---重启网络服务
    # systemctl stop network
    # systemctl start network

    二、设定 vm.min_free_kbytes 参数为物理内存的0.4%

    For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1073742 ***

    ---此次修改主机内存 512G ,所以 这个值是 1073742*2=2147484

    vm.min_free_kbytes=2147484

    [oracle@test1 ~]$ cat /proc/sys/vm/min_free_kbytes
    65536
    [oracle@test1 ~]$

    调整MIN_FREE_KBYTES的目的是保持物理内存有足够的空闲空间,防止突发性的换页。

    vi /etc/sysctl.conf
    vm.min_free_kbytes=2147484

    --使生效
    sysctl -p

  • 相关阅读:
    编译使用tinyxml
    GitLab 项目创建后地址由Localhost改为实际IP的方法
    树莓派相机设定
    MongoDB的数据备份与恢复
    Nginx PHP fpm forbidden 原因
    PSR2规范
    docker 日志管理
    Docker 拷贝文件
    Docker MySQL基本操作
    deepin安装php5.6
  • 原文地址:https://www.cnblogs.com/ss-33/p/11057478.html
Copyright © 2020-2023  润新知