双节点RAC环境，数据库 racdb 实例1：racdb1 实例2：racdb2
1.IP规划
名称 oracle-db1 oracle-db2
PUBLIC IP 10.10.0.17 10.10.0.18
Virtual IP 10.10.0.8 10.10.0.9
Private IP 10.0.0.1 10.0.0.2
SCAN IP 10.10.0.10
SID racdb1 racdb2
2.两个节点安装系统补丁（64位）
binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3（32 位）
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-2.5-24（32 位）
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5（32 位）
glibc-headers-2.5
libaio-0.3.106
libaio-0.3.106（32 位）
libaio-devel-0.3.106
libaio-devel-0.3.106（32 位）
libgcc-4.1.2
libgcc-4.1.2（32 位）
libstdc++-4.1.2
libstdc++-4.1.2（32 位）
libstdc++-devel 4.1.2
make-3.81
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-2.2.11（32 位）
unixODBC-devel-2.2.11
unixODBC-devel-2.2.11（32 位）
[root@oracle-db1 ~]# yum install -y binutils* compat-libstdc* elfutils-libelf* gcc* glibc* libaio* libgcc* libstdc++* make* sysstat* unixODBC*
[root@oracle-db2 ~]#yum install -y binutils* compat-libstdc* elfutils-libelf* gcc* glibc* libaio* libgcc* libstdc++* make* sysstat* unixODBC*
3两个节点.网络配置
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public IP
10.10.0.17 oracle-db1
10.10.0.18 oracle-db2
# Private IP
10.0.0.1 db1-priv
10.0.0.2 db2-priv
# Virtual IP
10.10.0.8 db1-vip
10.10.0.9 db2-vip
# Scan IP
10.10.0.10 rac-scan
4.两个节点服务器停用NTP时间同步
[root@oracle-db1 ~]#sbin/service ntpd stop
[root@oracle-db1 ~]#chkconfig ntpd off
[root@oracle-db1 ~]#mv /etc/ntp.conf /etc/ntp.conf.org
[root@oracle-db1 ~]#rm /var/run/ntpd.pid
5.禁止两个节点所有防火墙
[root@oracle-db1 ~]#service iptables stop
[root@oracle-db1 ~]#service ip6tables stop
[root@oracle-db1 ~]#chkconfig iptables off
[root@oracle-db1 ~]#chkconfig ip6tables off
6.关闭所有节点服务器的selinux
[root@oracle-db1 ~]# vi /etc/sysconfig/selinux
#SELINUX=enforcing
SELINUX=disabled
7.给两个节点创建用户，组及目录，并赋予相应的权限
[root@oracle-db1 ~]#groupadd -g 1000 oinstall
[root@oracle-db1 ~]#groupadd -g 1200 asmadmin
[root@oracle-db1 ~]#groupadd -g 1201 asmdba
[root@oracle-db1 ~]#groupadd -g 1202 asmoper
[root@oracle-db1 ~]#groupadd -g 1300 dba
[root@oracle-db1 ~]#groupadd -g 1301 oper
[root@oracle-db1 ~]#useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid grid
[root@oracle-db1 ~]#useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle oracle
[root@oracle-db1 ~]#mkdir -p /home/u01/app/grid
[root@oracle-db1 ~]#mkdir -p /home/u01/app/11.2.0/grid
[root@oracle-db1 ~]#chown -R grid:oinstall /home/u01
[root@oracle-db1 ~]#mkdir -p /home/u01/app/oracle
[root@oracle-db1 ~]#chown oracle:oinstall /home/u01/app/oracle
[root@oracle-db1 ~]#chmod -R 775 /home/u01
[root@oracle-db1 ~]#passwd oracle
[root@oracle-db1 ~]#passwd grid
[root@oracle-db1 ~]# id oracle -a
uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)
[root@oracle-db1 ~]# id grid -a
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper),1300(dba)
8.修改两个节点的系统参数
在“/etc/sysctl.conf”文件中添加如下内容
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
编辑完成后，执行下面命令生效：
#/sbin/sysctl -p
9.在/etc/security/limits.conf添加如下内容
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
10.在 /etc/pam.d/login添加
session required pam_limits.so
11.给两个节点的grid,oracle用户配置环境变量
[root@oracle-db1 ~]# su - oracle
[oracle@oracle-db1 ~]$ vi .bash_profile
ORACLE_HOSTNAME=oracle-db1; export ORACLE_HOSTNAME
ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME
ORACLE_SID=racdb1; export ORACLE_SID
JAVA_HOME=/usr/local/java: export JAVA_HOME
ORACLE_BASE=/home/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME
ORACLE_PATH=/home/u01/app/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/home/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
[oracle@oracle-db1 ~]$ su - grid
[grid@oracle-db1 ~]$ vi .bash_profile
ORACLE_SID=+ASM1; export ORACLE_SID
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_BASE=/home/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/home/u01/app/11.2.0/grid; export ORACLE_HOME
ORACLE_PATH=/home/u01/app/oracle/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/home/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
在oracle-db2上需要将oracle用户的ORACLE_SID改为racdb2,grid用户的ORACLE_SID改为+ASM2
12配置共享磁盘udev
查看多链路劲的存储信息
[root@oracle-db1 ~]# multipath -ll
VOTE3 (360060e80122d0b0050402d0b00000241) dm-7 HITACHI,OPEN-V
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:6 sdi 8:128 active ready running
`- 2:0:2:6 sds 65:32 active ready running
VOTE2 (360060e80122d0b0050402d0b00000240) dm-8 HITACHI,OPEN-V
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:5 sdh 8:112 active ready running
`- 2:0:2:5 sdr 65:16 active ready running
VOTE1 (360060e80122d0b0050402d0b0000023f) dm-6 HITACHI,OPEN-V
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:4 sdg 8:96 active ready running
`- 2:0:2:4 sdq 65:0 active ready running
DATA4 (360060e80122d0b0050402d0b0000023e) dm-4 HITACHI,OPEN-V
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:3 sdf 8:80 active ready running
`- 2:0:2:3 sdp 8:240 active ready running
DATA3 (360060e80122d0b0050402d0b0000023d) dm-5 HITACHI,OPEN-V
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:2 sde 8:64 active ready running
`- 2:0:2:2 sdo 8:224 active ready running
DATA2 (360060e80122d0b0050402d0b0000023c) dm-3 HITACHI,OPEN-V
size=310G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:1 sdd 8:48 active ready running
`- 2:0:2:1 sdn 8:208 active ready running
DATA1 (360060e80122d0b0050402d0b0000023b) dm-2 HITACHI,OPEN-V
size=310G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:3:0 sdc 8:32 active ready running
`- 2:0:2:0 sdm 8:192 active ready running
1) 在两台服务器上编辑/etc/udev/rules.d/50-udev.rules，添加如下内容：
SUBSYSTEM=="block", KERNEL=="dm-2", GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-3", GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-4, GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-5", GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-6", GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-7", GROUP="asmadmin", OWNER="grid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="dm-8", GROUP="asmadmin", OWNER="grid", MODE="0660"
也可以写成
SUBSYSTEM=="block", KERNEL=="dm-[2-8]", GROUP="asmadmin", OWNER="grid", MODE="0660"
2) 在两台服务器上执行下面命令，重启udev
[root@oracle-db1 ~]#/sbin/start_udev
13.安装grid软件
两个节点安装grid下的包
解压进到该目录
[root@oracle-db1 ~]# cd /home/software/grid
[root@oracle-db1 grid]# cd rpm/
[root@oracle-db1 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing... ########################################### [100%]
Using default group oinstall to install package
1:cvuqdisk ########################################### [100%]
[grid@oracle-db1 grid]$ xhost +
access control disabled, clients can connect from any host
[grid@oracle-db1 grid]$ ./runInstaller
在root用户执行上述弹出的脚本，要按先后顺序
[root@oracle-db1 ~]# /home/u01/app/oraInventory/orainstRoot.sh
Changing permissions of /home/u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /home/u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oracle-db1 ~]# /home/u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'oracle-db1'
CRS-2676: Start of 'ora.mdnsd' on 'oracle-db1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oracle-db1'
CRS-2676: Start of 'ora.gpnpd' on 'oracle-db1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oracle-db1'
CRS-2672: Attempting to start 'ora.gipcd' on 'oracle-db1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oracle-db1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'oracle-db1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oracle-db1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oracle-db1'
CRS-2676: Start of 'ora.diskmon' on 'oracle-db1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oracle-db1' succeeded
ASM created and started successfully.
Disk Group CRS created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 956d6cc946fc4f75bfdabd0bb28afd97.
Successful addition of voting disk 9d7f8b1ba3b34f4bbf89c41eabb7969a.
Successful addition of voting disk 7e81e9e8a00a4fa1bffd65fa3d7069c0.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 956d6cc946fc4f75bfdabd0bb28afd97 (/dev/mapper/VOTE1) [CRS]
2. ONLINE 9d7f8b1ba3b34f4bbf89c41eabb7969a (/dev/mapper/VOTE2) [CRS]
3. ONLINE 7e81e9e8a00a4fa1bffd65fa3d7069c0 (/dev/mapper/VOTE3) [CRS]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'oracle-db1'
CRS-2676: Start of 'ora.asm' on 'oracle-db1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'oracle-db1'
CRS-2676: Start of 'ora.CRS.dg' on 'oracle-db1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@oracle-db2 ~]# /home/u01/app/oraInventory/orainstRoot.sh
Changing permissions of /home/u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /home/u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oracle-db2 ~]# /home/u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402:
The CSS daemon was started in exclusive mode but found an active CSS
daemon on node oracle-db1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
14.grid用户创建ASM磁盘
[grid@oracle-db1 ~]$ asmca
15.安装Oracle11g软件
Error in invoking target 'install' of makefile '/u01/app/oracle/product/dbhome_1/ctx/lib/ins_ctx.mk'.
如果安装时弹出该错误，什么系统包没有装完整，可能是32的没有装好，64位的软件要包含安装32位的系统包
yum install -y glibc-devel*
到94%时会弹出脚本框执行相应的脚本，按顺序执行
[root@oracle-db1 ~]# /home/u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /home/u01/app/oracle/product/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@oracle-db2 ~]# /home/u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /home/u01/app/oracle/product/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
16.创建数据库实例
