客户网络调整需要更改私有网络，数据库版本为19c ，集群使用flex asm
查看是否使用flex asm
[+ASM2]@ht02[/home/grid]$asmcmd showclustermode
ASM cluster : Flex mode enabled - Direct Storage Access
更改前私网配置
[+ASM2]@ht02[/home/grid]$oifcfg getif enp0s8 192.168.20.0 global public enp0s9 192.168.0.0 global cluster_interconnect,asm [+ASM2]@ht02[/home/grid]$oifcfg iflist enp0s3 10.0.2.0 enp0s8 192.168.20.0 enp0s9 192.168.0.0 enp0s10 192.168.1.0 enp0s10 169.254.0.0
更改私有网络
/u01/app/grid/bin/oifcfg setif -global enp0s10/192.168.1.0:cluster_interconnect,asm /u01/app/grid/bin/oifcfg delif -global enp0s9
查看私网配置
[+ASM2]@ht02[/home/grid]$oifcfg getif enp0s8 192.168.20.0 global public enp0s10 192.168.1.0 global cluster_interconnect,asm
重启主机或者重启集群后发现只有一个节点能启动，另一个节点一直卡在启动ora.storage
[+ASM1]@ht01[/home/grid]$crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE ht01 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE ht01 STABLE
ora.crf
1 ONLINE ONLINE ht01 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE ht01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE ht01 STABLE
ora.ctssd
1 ONLINE ONLINE ht01 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE ht01 STABLE
ora.gipcd
1 ONLINE ONLINE ht01 STABLE
ora.gpnpd
1 ONLINE ONLINE ht01 STABLE
ora.mdnsd
1 ONLINE ONLINE ht01 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
查看日志如下
grid alert日志 2022-07-29 09:46:08.994+CRS-5006: Unable to automatically select a network interface which has subnet mask and subnet number 192.168.0.0 . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/diag/crs/ht01/crs/trace/crsd_orarootagent_root.trc".
crsd_orarootagent_root.trc
2022-07-29 09:45:58.166 :CLSDYNAM:723580672: [ora.asmnet1.asmnetwork]{1:4936:2} [check] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.asmnet1.asmnetwork check" encountered the following error:
CRS-5006: Unable to automatically select a network interface which has subnet mask and subnet number 192.168.0.0
. For details refer to "(:CLSN00109:)" in "/u01/app/11.2.0/grid/diag/crs/ht01/crs/trace/crsd_orarootagent_root.trc"
查看asmnetwork配置
[+ASM2]@ht02[/home/grid]$srvctl config asmnetwork ASM network 1 exists Subnet IPv4: 192.168.0.0// Subnet IPv6: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes:
修改asmnetwork
[root@ht01 ~]# /u01/app/grid/bin/srvctl remove asmnetwork -netnum 1 -force [root@ht01 ~]# /u01/app/grid/bin/srvctl add asmnetwork -subnet 192.168.1.0 [root@ht01 ~]# /u01/app/grid/bin/srvctl config asmnetwork ASM network 1 exists Subnet IPv4: 192.168.1.0// Subnet IPv6: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes:
在重启集群发现节点1ora.storage还是启动不起来，查看资源属性
[+ASM2]@ht02[/home/grid]$crsctl stat res -p|grep SUBNET SUBNET=192.168.0.0 REGISTRATION_INVITED_SUBNETS= REGISTRATION_INVITED_SUBNETS= REGISTRATION_INVITED_SUBNETS= USR_ORA_SUBNET=192.168.1.0 USR_ORA_SUBNET=192.168.20.0
集群里还是有192.168.0.0的网段，需要修改成192.168.1.0，此时alert日志报错如下
2019-08-13 19:48:27.006 : USRTHRD:893126400: {0:5:3} 8154 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2019-08-13 19:48:27.015 : USRTHRD:893126400: {0:5:3} 8154 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2019-08-13 19:48:27.084 : CLSCRED:893126400: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.303a14fb7cd04f7affd9f5ed7ebdcd83.root not found
2019-08-13 19:48:27.084 : USRTHRD:893126400: {0:5:3} 7872 Error 4 opening dom root in 0x7fd00820c3d0
2019-08-13 19:48:30.691 : USRTHRD:893126400: {0:5:3} ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
ORA-15077: could not locate ASM instance serving a required diskgroup
需要修改asm监听，查看当前asm监听
[+ASM1]@ht01[/home/grid]$srvctl config listener -asmlistener Name: ASMNET1LSNR_ASM Type: ASM Listener Owner: grid Subnet: 192.168.0.0 Home: <CRS home> End points: TCP:1525 Listener is enabled. Listener is individually enabled on nodes: Listener is individually disabled on nodes:
修改asm监听及查看
[+ASM2]@ht02[/home/grid]$srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force [+ASM2]@ht02[/home/grid]$srvctl config listener -asmlistener PRCN-2044 : No listener exists [+ASM1]@ht01[/home/grid]$srvctl add listener -listener ASMNET1LSNR_ASM -oraclehome $ORACLE_HOME -asmlistener -netnum 1 [+ASM1]@ht01[/home/grid]$srvctl config listener -asmlistener Name: ASMNET1LSNR_ASM Type: ASM Listener Owner: grid Subnet: 192.168.1.0 Home: <CRS home> End points: TCP:1525 Listener is enabled. Listener is individually enabled on nodes: Listener is individually disabled on nodes:
查看集群属性
[+ASM2]@ht02[/home/grid]$crsctl stat res -p|grep SUBNET SUBNET=192.168.1.0 REGISTRATION_INVITED_SUBNETS= REGISTRATION_INVITED_SUBNETS= REGISTRATION_INVITED_SUBNETS= USR_ORA_SUBNET=192.168.1.0 USR_ORA_SUBNET=192.168.20.0
重启grid，正常启动
查看asm监听
[+ASM2]@ht02[/home/grid]$lsnrctl status ASMNET1LSNR_ASM LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 29-JUL-2022 15:13:14 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM))) STATUS of the LISTENER ------------------------ Alias ASMNET1LSNR_ASM Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 29-JUL-2022 11:33:03 Uptime 0 days 3 hr. 40 min. 11 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/grid/network/admin/listener.ora Listener Log File /u01/app/11.2.0/grid/diag/tnslsnr/ht02/asmnet1lsnr_asm/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=ASMNET1LSNR_ASM))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.11)(PORT=1525))) Services Summary... Service "+ASM" has 2 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Instance "+ASM2", status READY, has 2 handler(s) for this service... Service "+ASM_ARCHDG" has 2 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Instance "+ASM2", status READY, has 2 handler(s) for this service... Service "+ASM_DATADG" has 2 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Instance "+ASM2", status READY, has 2 handler(s) for this service... Service "+ASM_OCR" has 2 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Instance "+ASM2", status READY, has 2 handler(s) for this service...
总结：12c以后修改私网ip，还须修改asmnetwork、ASMNET1LSNR_ASM，参考mos How to Modify ASM Network And ASM Listener Information in Grid Infrastructure (Doc ID 2856306.1)