安装介质下载
打开firefox,输入:https://edelivery.oracle.com
点击"Sign In",输入帐号、密码,登陆edelivery网站。
在虚拟机中安装Oracle Linux 6.6操作系统,整个操作系统的安装过程比较简单,只需要注意的是在安装过程中选上"软件开发"即可。
安装系统包
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-oracle
# yum install -y lm_sensors net-snmp net-snmp-utils perl-Compress-Zlib perl-HTML-Parser perl-libwww-perl perl-HTML-Tagset
系统网络配置
[root@cell03 ~]# more /etc/sysconfig/network-scripts/ifcfg-eth0
UUID=997505ac-4d9a-47cc-a5ae-452c9f185fcb
[root@cell03 ~]# more /etc/sysconfig/network-scripts/ifcfg-eth1
UUID=0a4024ae-8ae7-480e-8a7f-2ddd8ae0db59
[root@cell03 ~]# more /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.162 dmdb02.oracle.com dmdb02
10.0.0.12 dmdb02-priv.oracle.com dmdb02-priv
192.168.56.165 cell03.oracle.com cell03
10.0.0.15 cell03-priv.oracle.com cell03-priv
创建目录
配置系统内核
# vi /etc/security/limits.conf
session required pam_limits.so
配置RDS协议
install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma
关闭防火墙和selinux
同时,修改/etc/selinux/config 文件,将SELINUX=enforcing改为SELINUX=disabled
安装Cell软件
从下载的存储软件V790358-01.zip中,可以解压出nfsimg-12.1.2.3.3-161013-DL180.tar包,进一步解压,会生成cellbits/cellrpms.tbz文件,该文件其实是一个ZIP包,对该文件继续解压,会解压出jdk1.8.0_66-1.8.0_66-fcs.x86_64.rpm.
[root@cell03 tmp]# rpm -ihv jdk1.8.0_66-1.8.0_66-fcs.x86_64.rpm
warning: jdk1.8.0_66-1.8.0_66-fcs.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY
Preparing... ########################################### [100%]
1:jdk1.8.0_66 ########################################### [100%]
[root@cell03 tmp]# unzip cell.bin
warning [cell.bin]: 25582 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-12.1.2.3.3_LINUX.X64_161013-1.x86_64.rpm
[root@cell03 tmp]# rpm -ihv cell-12.1.2.3.3_LINUX.X64_161013-1.x86_64.rpm
perl(XML::Simple) is needed by cell-12.1.2.3.3_LINUX.X64_161013-1.x86_64
[root@cell03 tmp]# rpm -ihv perl-XML-Simple-2.18-6.el6.noarch.rpm
Preparing... ########################################### [100%]
1:perl-XML-Simple ########################################### [100%]
[root@cell03 tmp]# rpm -ihv cell-12.1.2.3.3_LINUX.X64_161013-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
Host name from hostname comamnd is cell03.oracle.com
Host name to use: cell03.oracle.com
1:cell ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell12.1.2.3.3_LINUX.X64_161013/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell12.1.2.3.3_LINUX.X64_161013/cellsrv/deploy/log directory
/opt/oracle/cell12.1.2.3.3_LINUX.X64_161013/cellsrv/deploy
Logout and then re-login to use the new cell environment.
调试Cell软件
安装完cell软件,在后续启动cell服务的过程中会出现如下一些错误:
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
The STARTUP of CELLSRV services was not successful.
CELL-01531: Unable to parse the cellinit.ora file due to incorrect parameters in the file.
The STARTUP of MS services was successful.
检查cellinit.ora文件,发现该文件内容为空,可以手动修改该文件,添加如下内容:
CELL host name=cell03.oracle.com
CELL version=12.1.2.3.3,label=OSS_12.1.2.3.3_LINUX.X64_161013,Fri_Oct_14_02:57:46_PDT_2016
CELLSRV version md5: f13842609a16ff8687fafcec297fc915
OS Stats: Physical memory: 3696 MB. Num cores: 2
CELLSRV configuration parameters:
Cellsrv max memory not set. Total available mem: 3696 MB is less than required minimum: 3891 MB.
Running on simulated hardware in production environment
ocl_si_extent_desc_alloc: Extent descriptor allocation failed. Error:3 (Invalid parameters)
ocl_si_cache_config: SI alloc extent desc failed. num extent:0
System state dump completed (with 1 failures)
CELLSRV error - ORA-600 internal error
存储节点虚拟机分配的物理内存太小了,增大虚拟机的物理内存,该问题会消失。
经过大量测试,给虚拟机分配的内存必须超过10000M,否则因为内存问题,除了以上错误之外,还会报各种不同的错误,这里不再列举其它错误信息。
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
The STARTUP of CELLSRV services was successful.
The STARTUP of MS services was not successful.
CELL-01554: MS startup failed for unknown reasons.
[RS] Start service MS failed with error: -74.
[RS] Service MS is no longer present
RS-7445 [Serv MS is absent] [It will be restarted] [] [] [] [] [] [] [] [] [] []
修改/opt/oracle/cell/cellsrv/deploy/msdomain/bin/setDomainEnv.sh文件:
JAVA_OPTIONS="-XX:-UseLargePages -XX:ParallelGCThreads=8 ........(内容太多,略) .......-Dweblogic.ListenPort=8888 -Djava.library.path=/opt/oracle/cell12.1.2.3.3_LINUX.X64_161013/cellsrv/lib -Djava.security.egd=file:/dev/./urandom"
[RS] Start service CELLSRV failed with error: -63.
RSBK version=12.1.2.3.3,label=OSS_12.1.2.3.3_LINUX.X64_161013,Fri_Oct_14_02:57:42_PDT_2016
[RS] Started Service RS_BACKUP with pid 10118
[RS] Kill previous monitoring process for core RS
这个故障是由于防火墙未关闭。 OEL6存在一个问题,即chkconfig iptables off 无法关闭防火墙的自动启动功能,即使执行了该命令,当操作系统重启后,还是会自动启动防火墙。
[root@cell03 ~]# cellcli -e list cell detail
cellVersion: OSS_12.1.2.3.3_LINUX.X64_161013
id: c3043721-8031-4541-8d35-88a7ac489529
kernelVersion: 3.8.13-44.1.1.el6uek.x86_64
Issuer CN=cell03.oracle.com,OU=Oracle Exadata,O=Oracle Corporation,L=Redwood City,ST=California,C=US
可见,cellsrv、MS、RS三个服务都是running状态,说明cell存储软件在虚拟机上配置成功。
建立虚拟的磁盘和闪盘
# mkdir -p /opt/oracle/cell/disks/raw
# cd /opt/oracle/cell/disks/raw
dd if=/dev/zero of=disk01 bs=1M count=1024
dd if=/dev/zero of=disk02 bs=1M count=1024
dd if=/dev/zero of=disk03 bs=1M count=1024
dd if=/dev/zero of=disk04 bs=1M count=1024
dd if=/dev/zero of=disk05 bs=1M count=1024
dd if=/dev/zero of=disk06 bs=1M count=1024
dd if=/dev/zero of=disk07 bs=1M count=1024
dd if=/dev/zero of=disk08 bs=1M count=1024
dd if=/dev/zero of=disk09 bs=1M count=1024
dd if=/dev/zero of=disk10 bs=1M count=1024
dd if=/dev/zero of=disk11 bs=1M count=1024
dd if=/dev/zero of=disk12 bs=1M count=1024
dd if=/dev/zero of=FLASH01 bs=1M count=1024
dd if=/dev/zero of=FLASH02 bs=1M count=1024
dd if=/dev/zero of=FLASH03 bs=1M count=1024
dd if=/dev/zero of=FLASH04 bs=1M count=1024
[root@cell03 raw]# chmod +x dd.sh
[root@cell03 raw]# rm -rf dd.sh
-rw-r--r-- 1 root root 1073741824 Jan 6 15:28 disk01
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk02
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk03
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk04
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk05
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk06
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk07
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk08
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk09
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk10
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk11
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 disk12
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 FLASH01
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 FLASH02
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 FLASH03
-rw-r--r-- 1 root root 1073741824 Jan 6 15:29 FLASH04
执行dd.sh脚本,创建对应的磁盘和闪盘:其中磁盘12块,每块大小为1GB,闪盘4块,每块大小也是1GB。
创建celldisk
CellCLI: Release 12.1.2.3.3 - Production on Fri Jan 06 15:33:57 CST 2017
Copyright (c) 2007, 2016, Oracle. All rights reserved.
CellDisk FD_00_cell03 successfully created
CellDisk FD_01_cell03 successfully created
CellDisk FD_02_cell03 successfully created
CellDisk FD_03_cell03 successfully created
CellDisk CD_disk01_cell03 successfully created
CellDisk CD_disk02_cell03 successfully created
CellDisk CD_disk03_cell03 successfully created
CellDisk CD_disk04_cell03 successfully created
CellDisk CD_disk05_cell03 successfully created
CellDisk CD_disk06_cell03 successfully created
CellDisk CD_disk07_cell03 successfully created
CellDisk CD_disk08_cell03 successfully created
CellDisk CD_disk09_cell03 successfully created
CellDisk CD_disk10_cell03 successfully created
CellDisk CD_disk11_cell03 successfully created
CellDisk CD_disk12_cell03 successfully created
创建flashlog和flashcache
Flash log cell03_FLASHLOG successfully created
CellCLI> create flashcache all
Flash cache cell03_FLASHCACHE successfully created
CellCLI> list flashcache detail
cellDisk: FD_01_cell03,FD_00_cell03,FD_02_cell03,FD_03_cell03
creationTime: 2017-01-06T15:38:12+08:00
id: 5fb44bac-a28a-4881-ba0f-09ff6b8fe023
cellDisk: FD_01_cell03,FD_02_cell03,FD_00_cell03,FD_03_cell03
creationTime: 2017-01-06T15:37:48+08:00
id: 61e9b2fe-8673-4b1f-99f2-aa74b9321490
创建griddisk
CellCLI> create griddisk all harddisk prefix=DATA, size=700M
GridDisk DATA_CD_disk01_cell03 successfully created
GridDisk DATA_CD_disk02_cell03 successfully created
GridDisk DATA_CD_disk03_cell03 successfully created
GridDisk DATA_CD_disk04_cell03 successfully created
GridDisk DATA_CD_disk05_cell03 successfully created
GridDisk DATA_CD_disk06_cell03 successfully created
GridDisk DATA_CD_disk07_cell03 successfully created
GridDisk DATA_CD_disk08_cell03 successfully created
GridDisk DATA_CD_disk09_cell03 successfully created
GridDisk DATA_CD_disk10_cell03 successfully created
GridDisk DATA_CD_disk11_cell03 successfully created
GridDisk DATA_CD_disk12_cell03 successfully created
CellCLI> create griddisk all harddisk prefix=RECO
GridDisk RECO_CD_disk01_cell03 successfully created
GridDisk RECO_CD_disk02_cell03 successfully created
GridDisk RECO_CD_disk03_cell03 successfully created
GridDisk RECO_CD_disk04_cell03 successfully created
GridDisk RECO_CD_disk05_cell03 successfully created
GridDisk RECO_CD_disk06_cell03 successfully created
GridDisk RECO_CD_disk07_cell03 successfully created
GridDisk RECO_CD_disk08_cell03 successfully created
GridDisk RECO_CD_disk09_cell03 successfully created
GridDisk RECO_CD_disk10_cell03 successfully created
GridDisk RECO_CD_disk11_cell03 successfully created
GridDisk RECO_CD_disk12_cell03 successfully created
CellCLI> list griddisk RECO_CD_disk12_cell03 detail
creationTime: 2017-01-16T00:03:09+08:00