Oracle RAC 11gR2 on OEL6

Let's begin 2015 by trying to detail an Oracle Real Application Cluster 11G Release 2 Installation on Oracle Enterprise Linux 6 (the update used during this install is 6.4 , but this will most probably works with other OEL6 update). 
To make things easier (and more readable), I've divided this post in 04 main sections. The first section is just about Physical Installation, the second is about OS Installation and Pre-Installation Steps, the third one will deal with Grid Infrastructure Software Installation and the Fourth is about RDBMS Software Installation.

I. Physical Installation:

This very short section is just about the description of Physical Installation performed for this Installation. As a picture is worth a thousands words, let's start by showing a small picture of the Infrastructure we are deploying.

So, we've 02 Physical Nodes with 04 Ethernet Ports for each nodes. Two of these Ethernet Ports will be slaves of Bond0 (active-passive mode) and connected to a Public Network while the others two ports will be initially configured as Independent Interfaces and connected to a Private Network. In fact, Rather than bonding these private NICs, we leave them both configured with static fixed addresses and during the installation, we'll specify both interfaces as private. Grid Infrastructure will assign a virtual IP address in 169.245.x.x network to these Interfaces and use HAIP protocol. The result is equivalent to Bonding in Load-balancing Mode.
For Storage connectivity, we have 02 FC Ports connected on a Storage Area Network, Shared Disks that will be used later as ASM Devices are provided by a Shared Storage connected to this SAN.

To close this section, I have few words about Nodes Physical Requirement: The minimum required RAM is 2.5 GB. For systems with 2.5 GB to 16 GB RAM, Oracle recommends to use swap space equal to RAM. For systems with more than 16 GB RAM, use 0.75 x RAM as the swap space.

II. OS Installation and Pre-Installation Steps:

Having Physical Nodes Installation completed, we can move forward with the OS Installation and the Cluster Prerequisites Configuration.

II.I: OS Installation:

The Systems Installation is fully completed using Kickstart file.  I won't discuss the many advantages offered by Kickstart Installation, but will just highlight one of the main benefit I've found when using it for a cluster Installation (Not just Oracle RAC, but also RHEL Cluster, Linux-HA and other cluster frameworks). This advantage is the almost perfect symmetry between the clustered nodes (better make sure of that for a cluster Installation). There's also a large number of  configurations that can be completed using the %post section of Kickstart .  For instance, I've populated the /etc/hosts file and configured NTP in the %post section of the Kickstart File shown below.
In fact, I could have automated all the configurations that will be completed in the section II in this kickstart, but I've preferred to leave it like that in order to well describe these configurations in the next sections....

# Sample Kickstart file by Steve ESSO # Network Seetings for bond0 network --device=bond0 --noipv6 --bootproto=static --onboot=yes --hostname=dbnode01.stivesso.local --ip=192.168.0.11 --netmask=255.255.255.0 --gateway=192.168.0.1 --nameserver=192.168.0.5,192.168.0.6 --bondslaves=eth0,eth1 --bondopts=mode=active-backup,primary=eth0,miimon=80,updelay=60000 network --onboot yes --device eth2 --bootproto static --ip 10.252.40.41 --netmask 255.255.255.0 --noipv6 network --onboot yes --device eth3 --bootproto static --ip 10.252.40.43 --netmask 255.255.255.0 --noipv6 # Default Root Pw:# rootpw --iscrypted $6$16_CHARACTER_SAL$ti1MDnLMg9j8ahFnpRctlh7/koWcZdEXG4T/jdtTHCqXfwDrHClRU1NIYvQ4dR5GpI9fLtNgnZ1DfHeig74uq. # Do not use firewall by default firewall --disabled # SELinux configuration selinux --disabled # Type of auth configs authconfig --enableshadow --passalgo=sha512 # Install OS instead of upgrade install # Set Install Lang lang en_US.UTF-8 # Keyboard US-international keyboard us-acentos # Use URL to install installation media url --url=http://192.168.0.7/media_oel6u5 # User Text only mode. text # Do not configure the X Window System skipx # System timezone timezone --utc Africa/Douala # - DISK PARTITIONING # Clear all the partitions / disks clearpart --all # Clear the MBR. Helps avoid problems with MBR. zerombr # Got the /dev/sda Disk id using smartctl -i /dev/sda part /boot --fstype ext4 --size 512 --asprimary --ondisk=disk/by-id/scsi-3600508b1001c38df3408679c9c4e3a87 part pv.01 --size 1024 --grow --asprimary --ondisk=disk/by-id/scsi-3600508b1001c38df3408679c9c4e3a87 # LVM disks: volgroup VG1 pv.01 logvol /home --fstype=ext4 --name=lv_home --vgname=VG1 --size=5120 logvol / --fstype=ext4 --name=lv_root --vgname=VG1 --size=5120 logvol /usr --fstype=ext4 --name=lv_usr --vgname=VG1 --size=10240 logvol swap --name=lv_swap --vgname=VG1 --size=20480 logvol /tmp --fstype=ext4 --name=lv_tmp --vgname=VG1 --size=5120 logvol /var --fstype=ext4 --name=lv_var --vgname=VG1 --size=20480 logvol /u01 --fstype=ext4 --name=lv_u01 --vgname=VG1 --size=51200 # Boot Loader Location bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet vga=0x333" %packages #Note 1441282.1 used to add the packages for 11gR2 install @base @basic-desktop @client-mgmt-tools @core @debugging @desktop-debugging @desktop-platform @directory-client @fonts @general-desktop @graphical-admin-tools @input-methods @internet-browser @java-platform @legacy-x @network-file-system-client @perl-runtime @print-client @remote-desktop-clients @server-platform @x11 gcc gcc-c++ compat-libcap1 libstdc++-* compat-libstdc++-* ksh libaio-devel libaio xclock dos2unix lsscsi sg3_utils abrt-gui certmonger genisoimage krb5-workstation libXmu mtools oddjob pam_krb5 pax perl-DBD-SQLite python-dmidecode sgpio wodim system-config-lvm glibc glibc-devel libXp*.el6 #32bits for runInstaller glibc.i686 libXp*.i686 glibc-devel*.i686 compat-libstdc++-296.i686 libstdc++*.i686 libaio.i686 libaio-devel*.i686 device-mapper-multipath %end ########################################## # Post script: ########################################## %post # Populate hosts file cat <<EOF1 >/etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Public IP 192.168.0.11 dbnode01 dbnode01.stivesso.local 192.168.0.12 dbnode02 dbnode02.stivesso.local #VIP 192.168.0.15 dbnode01-vip dbnode01-vip.stivesso.local 192.168.0.16 dbnode02-vip dbnode02-vip.stivesso.local #Private Network 192.168.1.41 dbnode01-prv1 192.168.1.43 dbnode01-prv2 192.168.1.42 dbnode02-prv1 192.168.1.44 dbnode02-prv2 EOF1 # Configure NTP cat <<EOF2 >/etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g -x" EOF2 %end

II.2: Network Configuration:

As described in the first Section, we've 02 Physical Nodes with 04 Physical Ethernet Interfaces . For redundancy purpose, the first two network interfaces are slaves of Bond0 and will be used for public IP. Though we've configured the 02 Private Interfaces as Independent Interfaces, HAIP will automatically be configured during the Grid Infrastructure Installation. 

In fact, HAIP is enabled for the private network by selecting multiple interfaces to use as private interfaces during the Grid Infrastructure Installation. This functionality is available starting with Oracle Grid Infrastructure 11g Release 2(11.2.0.2). Though it's enabled by default, we still have to set IP on the interfaces and plumb/up these interfaces. In fact, Oracle Grid currently creates a Virtual  IP on the private network adapters using the 169.254.*.* subnet for the HAIP. The purpose of HAIP is to load balance across all active interconnect interfaces, and failover to other available interfaces if one of the existing private adapters becomes unresponsive.

Each node will need a Virtual IP, and we'll also need a set of 03 Scan IPs. Below is a table with Details about this IP Address scheme.

Node Name	Public IP / Interfaces	Private IP Interface (HAIP)	Public Virtual IP
dbnode01	192.168.0.11 (bond0)	192.168.1.41 (eth2)/ 192.168.1.43 (eth3)	192.168.0.15
dbnode02	192.168.0.12 (bond0)	192.168.1.42 (eth2)/ 192.168.1.44 (eth3)	192.168.0.16

Scan Name
dbnode-scan.stivesso.local	192.168.0.17	192.168.0.18	192.168.0.19

Public and Private IPs were configured during the System Installation, with Public Interfaces being bonded as bond0 while private Interfaces were configured as Independent Interfaces. Below, we can see the output of the configuration applied by the KickStart configuration on these Interfaces.

[root@dbnode01 ~]# cd /etc/sysconfig/network-scripts/ [root@dbnode01 network-scripts]# cat ifcfg-bond0 DEVICE="bond0" BONDING_OPTS="mode=active-backup primary=eth0 miimon=80 updelay=60000" BOOTPROTO="static" DNS1="192.168.0.5" DNS2="192.168.0.6" GATEWAY="192.168.0.1" HOSTNAME="dbnode01.mtncameroon.net" IPADDR="192.168.0.11" IPV6INIT="no" MTU="1500" NETMASK="255.255.255.0" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Bond" UUID="532ecbce-9d07-4560-a7a4-205e2573af82" [root@dbnode01 network-scripts]# cat ifcfg-eth2 DEVICE="eth2" BOOTPROTO="static" HWADDR="A0:D3:C1:FB:6F:B6" IPADDR="192.168.1.41" IPV6INIT="no" NETMASK="255.255.255.0" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" UUID="d73deeb4-c11c-4763-811c-1518da1b88e5" [root@dbnode01 network-scripts]# cat ifcfg-eth3 DEVICE="eth3" BOOTPROTO="static" HWADDR="A0:D3:C1:FB:6F:B7" IPADDR="192.168.1.43" IPV6INIT="no" NETMASK="255.255.255.0" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" UUID="446b0cd2-e59e-4fa1-a99f-105933d07b43"  

The Public/Private and VIP with names associated  are all added to the DNS  and to the systems hosts files (see hosts file below, populated during the Kickstart Installation), while The scan IP/names associations must only be added to the DNS.

***/etc/hosts***
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Public IP 192.168.0.11 dbnode01 dbnode01.stivesso.local 192.168.0.12 dbnode02 dbnode02.stivesso.local #VIP 192.168.0.15 dbnode01-vip dbnode01-vip.stivesso.local 192.168.0.16 dbnode02-vip dbnode02-vip.stivesso.local #Private Network 192.168.1.41 dbnode01-prv1 192.168.1.43 dbnode01-prv2 192.168.1.42 dbnode02-prv1 192.168.1.44 dbnode02-prv2  

***DNS resolver / Make sure than scan name can be resolved***
[esso_s@dbnode01 ~]$ nslookup dbnode-scan.stivesso.local Server: 192.168.0.5 Address: 192.168.0.5#53 Name: dbnode-scan.stivesso.local Address: 192.168.0.19 Name: dbnode-scan.stivesso.local Address: 192.168.0.17 Name: dbnode-scan.stivesso.local Address: 192.168.0.18

The other important Network Service to configure is the NTP as we must make sure that server's clocks are synchronized. For this case I use a set of Internal NTP Servers and modified just the server statement in /etc/ntp.conf (replaced the default NTP Servers addresses by my own)

[root@dbnode01 grid]# grep ^server /etc/ntp.conf server 192.168.0.8 server 192.168.0.9 [root@dbnode01 grid]# ntpdate 192.168.0.8 [root@dbnode01 grid]# chkconfig ntpd on [root@dbnode01 grid]# service ntpd start Starting ntpd: [ OK ]

Also, to make sure that I won't hit PRVF-5436 error related to NTP configuration,I had to add the -x option to ntpd configuration and restart the NTP daemon. That was also completed during the Kickstart Installation. Below is the file that was populated.

[root@dbnode01 rpm]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g -x" [root@dbnode01 rpm]# service ntpd restart Shutting down ntpd: [ OK ] Starting ntpd: [ OK ]

Finally, the following Kernel Parameters need to be set on Private Interfaces for all the nodes. That'll disabled the RPF. Reverse Path Filtering (RPF) is a security feature which allow the kernel to throw away packets if the reply of a packet may not go through the interface it was received on. 

[root@dbnode01 ~]# echo "net.ipv4.conf.eth2.rp_filter=2" >> /etc/sysctl.conf [root@dbnode01 ~]# echo "net.ipv4.conf.eth3.rp_filter=2" >> /etc/sysctl.conf [root@dbnode01 ~]# sysctl -p

II.3 User Accounts and System Parameters

Let's now complete the creation of required User accounts and Systems parameters setting. Note that Oracle recommends the usage of different users accounts for the installation of the Grid Infrastructure (GI) and the Oracle RDBMS home. Following this recommendation , I'm trying to have Grid Infrastructure owned by user 'grid' and Oracle RDBMS owned by user 'oracle'. 
Most of the pre-installation configuration tasks can be completed by using the Oracle RDBMS Server 11gR2 Pre-install RPM for Oracle Linux 6 x86_64 (64 Bit) architecture, available from the Unbreakable Linux Network, or via the Oracle Public Yum repository. This package replaces the previous Oracle Validated RPM.

The Oracle RDBMS Pre-install RPM does the following:

* Automatically installs any additional packages needed for installing Oracle Grid       Infrastructure and Oracle Database 11gR2 (11.2.0.3).

* Creates an oracle user, and creates the oraInventory (oinstall) and OSDBA (dba) groups for that user. For security purposes, this user has no password by default and cannot login remotely. To enable remote login, please set a password using the "passwd" tool.

* Sets and verifies sysctl.conf settings, system startup parameters, user limits, and driver parameters to the minimum acceptable values based on recommendations from the Oracle Database.


Below is the package Installation procedure details (on both nodes), using Oracle Public Yum repository.

[root@dbnode01 ~]# cd /etc/yum.repos.d [root@dbnode01]# wget http://public-yum.oracle.com/public-yum-ol6.repo --2014-09-30 21:37:27-- http://public-yum.oracle.com/public-yum-ol6.repo .............................. Saving to: â€public-yum-ol6.repo.1” 100%[================>] 4,233 --.-K/s in 0s 2014-09-30 21:37:27 (203 MB/s) - â€public-yum-ol6.repo.1” saved [4233/4233] [root@dbnode01 ]# yum install oracle-rdbms-server-11gR2-preinstall.x86_64 Loaded plugins: security ............................. Setting up Install Process Resolving Dependencies --> Running transaction check ............................. Dependencies Resolved .............................. Install 4 Package(s) Total download size: 973 k Installed size: 2.5 M Is this ok [y/N]: y Downloading Packages: .............................. Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : ksh-20120801-10.el6_5.11.x86_64 1/4 Installing : compat-libstdc++-33-3.2.3-69.el6.x86_64 2/4 Installing : compat-libcap1-1.10-1.x86_64 3/4 Installing : oracle-rdbms-server-11gR2-preinstall-1.0-9.el6.x86_64 4/4 Verifying : compat-libcap1-1.10-1.x86_64 1/4 Verifying : compat-libstdc++-33-3.2.3-69.el6.x86_64 2/4 Verifying : ksh-20120801-10.el6_5.11.x86_64 3/4 Verifying : oracle-rdbms-server-11gR2-preinstall-1.0-9.el6.x86_64 4/4 Installed: oracle-rdbms-server-11gR2-preinstall.x86_64 0:1.0-9.el6 Dependency Installed: compat-libcap1.x86_64 0:1.10-1 compat-libstdc++-33.x86_64 0:3.2.3-69.el6 ksh.x86_64 0:20120801-10.el6_5.11 Complete!

To complete with software Installation, I recommend to also install cvuqdisk which is coming with the Grid Software, it is located under the rpm folder of the Grid Installer Binaries (see below).

[root@dbnode01 ~]# cd dba_softs/11.2.0.3/linux_86_64/grid [root@dbnode01 grid]# ls doc install readme.html response rpm runcluvfy.sh runInstaller sshsetup stage welcome.html [root@@dbnode01 grid]# cd rpm/ [root@@dbnode01 rpm]# ls cvuqdisk-1.0.9-1.rpm [root@@dbnode01 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%]


As Oracle recommends different users for the installation of the Grid Infrastructure and the Oracle RDBMS home, the next task is to create the grid Users and required Groups manually. We're also settings the user limits for this newly created Grid User. So on every Cluster Node, let's run the following (on both nodes).

Note that the UID for the Oracle User created by Oracle-rdbms-server-11gR2-preinstall is 54321, so I choose the next UID for Grid (same for the groups)

[root@dbnode01]# /usr/sbin/groupadd -g 54323 oper [root@dbnode01]# /usr/sbin/groupadd -g 54324 asmadmin [root@dbnode01]# /usr/sbin/groupadd -g 54325 asmdba [root@dbnode01]# /usr/sbin/groupadd -g 54326 asmoper [root@dbnode01]# /usr/sbin/useradd -u 54322 -c "Oracle Grid Infrastructure Owner" -g oinstall -G asmadmin,asmdba,asmoper grid [root@dbnode01]# /usr/sbin/usermod -c "Oracle Databases Owner" -G dba,asmadmin,asmdba,asmoper oracle

Now, we've to modify the User Limits settings for grid User (can also adjust if needed). That was done by the installation of the oracle-rdbms-server-11gR2-preinstall Installation for Oracle User, below is  /etc/security/limits.conf after this package's installation.

# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024 oracle soft nofile 1024 # oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536 oracle hard nofile 65536 # oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 16384 # refer orabug15971421 for more info. oracle soft nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384 oracle hard nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB oracle soft stack 10240 # oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB oracle hard stack 32768

We're just modifying the file by adding Grid Settings as seen below:

# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024 oracle soft nofile 1024 grid soft nofile 1024 # oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536 oracle hard nofile 65536 grid hard nofile 65536 # oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 16384 # refer orabug15971421 for more info. oracle soft nproc 16384 grid soft nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384 oracle hard nproc 16384 grid hard nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB oracle soft stack 10240 grid soft stack 10240 # oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB oracle hard stack 32768 grid hard stack 32768

Having the Users/Groups configured, we can move forward with the creation of the required Directory for both Oracle and Grid Users. As seen in my kickstart file, I'm using lv_u01 (mounted on /u01) for Oracle Software Binaries. I'm using the script below to complete the creation of the required directories and setting of their permissions (see comments in the script for details about each directory)

[root@dbnode01]# cat /root/grid_oracle_directory.sh #!/bin/bash set -x # Oracle Inventory: mkdir -p /u01/app/oraInventory chown -R grid:oinstall /u01/app/oraInventory chmod -R 775 /u01/app/oraInventory # Oracle Base for RDBMS: # Also creating /u01/app/oracle/cfgtoollogs to ensure that dbca is well running for rdbms Installation. mkdir -p /u01/app/oracle mkdir /u01/app/oracle/cfgtoollogs chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle # Oracle Base for Grid Infrastructure: mkdir /u01/app/grid chown -R grid:oinstall /u01/app/grid chmod -R 775 /u01/app/grid # Oracle Home for Grid Infrastructure : # During installation, ownership of the entire path to the Grid home is changed # to be root (/u01, /u01/11.2.0 and /u01/11.2.0/grid). # If we do not create a unique path to the Grid home, then after the Grid install, # we can encounter permission errors for other installations, # including any existing installations under the same path. mkdir -p /u01/11.2.0/grid chown -R grid:oinstall /u01/11.2.0 chmod -R 775 /u01/11.2.0 # Oracle Home for RDBMS: mkdir -p /u01/app/oracle/product/11.2.0/db_1 chown -R oracle:oinstall /u01/app/oracle/product chmod -R 775 /u01/app/oracle/product # u01 permissions chown oracle:oinstall /u01 && chmod 755 /u01 chown oracle:oinstall /u01/app && chmod 755 /u01/app

[root@dbnode01]# chmod ug+x /root/grid_oracle_directory.sh [root@dbnode01]# /root/grid_oracle_directory.sh

The next task of this sub-section is to reflect the created directory structure to grid and oracle Users environment by setting some of their environment variables. Note that for Oracle User, I'm using the same parameters on all the nodes, while ORACLE_SID for grid user across nodes differs.

Bash Profile for Oracle User on the 02 nodes

[oracle@dbnode01 ~]$ cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH # Oracle settings export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 export PATH=$PATH:$ORACLE_HOME/bin

Bash Profile for grid User on Node1
[grid@dbnode01 ~]$ cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH # Oracle Settings export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/11.2.0/grid export ORACLE_SID=+ASM1 export PATH=$PATH:$ORACLE_HOME/bin

Bash Profile for grid User on Node2
[grid@dbnode02 ~]$ cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH # Oracle Settings export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/11.2.0/grid export ORACLE_SID=+ASM2 export PATH=$PATH:$ORACLE_HOME/bin  

Finally, we need to create password for both Oracle and Grid using passwd command (just showing this as an example, Create your own password in a more secure way:-) )

[root@dbnode01 ~]# echo "0racleEsso" | passwd --stdin oracle [root@dbnode01 ~]# echo "Gr1dEsso" | passwd --stdin grid


II.4 Shared Storage Multipath Configuration

As described in Section I, the shared Disks are sitting on an external storage, and are accessible through a Multipath Software. For this configuration, we're making use of DM-Multipath. But before getting to Multipath configuration, let's summarize in a table the Shared Physical LUNs and the role they're supposed to play. 

Share Devices	Multipath Name	Size	Description
	asm-ocr-disk1	1GB	Oracle RAC OCR Disk
	asm-ocr-disk2	1GB	Oracle RAC OCR Disk
	asm-ocr-disk3	1GB	Oracle RAC OCR Disk
	asm-data-disk1	500GB	ASM Data Disk

The first three devices are to be added in a dedicated Disk Group for OCR/Vote Disk while the last one will be used for DB Data Disk Group. Before moving with Multipath Configuration, let's just rescan the scsi Devices to make sure that all the allocated LUNs are present on the Systems (to do on all the Cluster Nodes).

[root@dbnode01 ~]# for host in `ls /sys/class/scsi_host` > do > echo "- - -" > /sys/class/scsi_host/$host/scan > done

We can now start the multipath configuration by using mpathconf to make the initial configuration easy. 
Note that most of the steps below are completed on one node. This node's Multipath configuration will then be replicated to the other by copying /etc/multipah.conf

Print the current Multipath configuration, by default it's disabled [root@dbnode01 ~]# mpathconf multipath is disabled find_multipaths is disabled user_friendly_names is enabled dm_multipath module is loaded multipathd is chkconfiged off

Enable and start the Multipath service
[root@dbnode01 ~]# mpathconf --enable --with_multipathd y Starting multipathd daemon: [ OK ] [root@dbnode01 ~]# mpathconf multipath is enabled find_multipaths is disabled user_friendly_names is enabled dm_multipath module is loaded multipathd is chkconfiged on


Put in place a very minimal configuration (just to see multipath device), the reload multipath and list devices in this configuration
[root@dbnode01 ~]# cat /etc/multipath.conf # Written by Steve ESSO # By default use friendly name defaults { user_friendly_names yes } [root@dbnode01 ~]# multipath -r [root@dbnode01 ~]# multipath -l mpath2 (360060160cd0e2600fa9803d894dae311) dm-7 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=active | `- 2:0:1:0 sdg 8:96 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:0:0 sdc 8:32 active undef running mpath1 (360060160cd0e2600ecbf6e2b95dae311) dm-6 DGC,RAID 5 size=500G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=active | `- 2:0:1:1 sdh 8:112 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:0:1 sdd 8:48 active undef running mpath0 (360060160cd0e26000a3833b894dae311) dm-5 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=active | `- 2:0:0:2 sde 8:64 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:1:2 sdi 8:128 active undef running mpath3 (360060160cd0e2600c6cca80a95dae311) dm-8 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=active | `- 2:0:0:3 sdf 8:80 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:1:3 sdj 8:144 active undef running

Let's rename these devices by giving them the more comprehensible names that are in the Disk table above. For that we're adding the following to multipath.conf and reload paths.
multipaths { multipath { wwid 360000970000298701299533030373841 alias asm-ocr-disk1 } multipath { wwid 360000970000298701299533030373842 alias asm-ocr-disk2 } multipath { wwid 360000970000298701299533030373843 alias asm-ocr-disk3 } multipath { wwid 360000970000298701299533030373741 alias asm-data-disk1 } } [root@dbnode01 ~]# multipath -r May 14 11:00:44 | 360060160cd0e26000a3833b894dae311: rename mpath0 to asm-crs-disk3 rename: asm-crs-disk3 (360060160cd0e26000a3833b894dae311) undef DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=1 status=undef | `- 2:0:0:2 sde 8:64 active ready running `-+- policy='round-robin 0' prio=0 status=undef `- 2:0:1:2 sdi 8:128 active ready running May 14 11:00:44 | 360060160cd0e2600ecbf6e2b95dae311: rename mpath1 to asm-data-disk1 rename: asm-data-disk1 (360060160cd0e2600ecbf6e2b95dae311) undef DGC,RAID 5 size=500G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=1 status=undef | `- 2:0:1:1 sdh 8:112 active ready running `-+- policy='round-robin 0' prio=0 status=undef `- 2:0:0:1 sdd 8:48 active ready running May 14 11:00:44 | 360060160cd0e2600fa9803d894dae311: rename mpath2 to asm-crs-disk2 rename: asm-crs-disk2 (360060160cd0e2600fa9803d894dae311) undef DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=1 status=undef | `- 2:0:1:0 sdg 8:96 active ready running `-+- policy='round-robin 0' prio=0 status=undef `- 2:0:0:0 sdc 8:32 active ready running May 14 11:00:44 | 360060160cd0e2600c6cca80a95dae311: rename mpath3 to asm-crs-disk1 rename: asm-crs-disk1 (360060160cd0e2600c6cca80a95dae311) undef DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=1 status=undef | `- 2:0:0:3 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=0 status=undef `- 2:0:1:3 sdj 8:144 active ready running [root@dbnode01 ~]# multipath -l asm-crs-disk3 (360060160cd0e26000a3833b894dae311) dm-5 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=enabled | `- 2:0:0:2 sde 8:64 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:1:2 sdi 8:128 active undef running asm-crs-disk2 (360060160cd0e2600fa9803d894dae311) dm-7 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=enabled | `- 2:0:1:0 sdg 8:96 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:0:0 sdc 8:32 active undef running asm-crs-disk1 (360060160cd0e2600c6cca80a95dae311) dm-8 DGC,RAID 5 size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=enabled | `- 2:0:0:3 sdf 8:80 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:1:3 sdj 8:144 active undef running asm-data-disk1 (360060160cd0e2600ecbf6e2b95dae311) dm-6 DGC,RAID 5 size=500G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=0 status=enabled | `- 2:0:1:1 sdh 8:112 active undef running `-+- policy='round-robin 0' prio=0 status=enabled `- 2:0:0:1 sdd 8:48 active undef running

Node2 can now be configured by replicating the Node1 configuration.
[root@dbnode01 ~]# scp /etc/multipath.conf dbnode02:/etc/ [root@dbnode02 ~]# mpathconf --enable --with_multipathd y Starting multipathd daemon: [ OK ] [root@dbnode02 ~]# multipath -r

To complete with this subsection, we've to create a single partition on the Multipath disks. This procedure is mostly completed on one Node, with the exception of kpartx commands which are used to make sure that the physical path for the created partition exist on the nodes. Below is the output for 01 disk , this must be done for the all the shared disks.

[root@dbnode01 ~]# fdisk /dev/mapper/asm-data-disk1 WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): p Disk /dev/mapper/asm-data-disk1: 536.9 GB, 536881397760 bytes 255 heads, 63 sectors/track, 65272 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0008ea1c Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-65272, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-65272, default 65272): Using default value 65272 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 22: Invalid argument. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. [root@dbnode01 ~]# kpartx -a /dev/mapper/asm-data-disk1

II.5 Shared Storage / Oracle ASMLIB

Now that we have our shared disks well configured under DM-Multipath, we can move forward with ASMLIB Configuration. Below is a small recap of what we're aiming to complete in this sub-section.

Physical Disk	ASMLIB Disk Name	ASM Disk Group	ASM Redundancy
/dev/mapper/asm-crs-disk1p1	OCR_DISK1	OCR	High redundancy
/dev/mapper/asm-crs-disk2p1	OCR_DISK2	OCR	High redundancy
/dev/mapper/asm-crs-disk3p1	OCR_DISK3	OCR	High redundancy
/dev/mapper/asm-data-disk1p1	DATA_DISK1	DATA	External redundancy

Before diving in technical matters, let's first describe what is Oracle ASMLIB.  It is an optional support library for the Automatic Storage Management feature of the Oracle Database and allows an Oracle Database using ASM more efficient and capable access to the disk groups it is using. 

The Oracle ASMLib kernel driver is now included in the Unbreakable Enterprise Kernel.  No driver package needs to be installed when using this kernel.  The oracleasm-support and oracleasmlib packages still need to be installed from ULN. While the driver and support tools are on the Oracle Linux installation media, the oracleasmlib RPM is not (this package allows Oracle to access the kernel driver).  
Oracleasm-support is available on the Installation Media and on the Public Yum repository, That isn't the case for oracleasmlib which is only available on ULN. Subscribers to ULN can use yum(8) or up2date(8) to download and install the package on their servers. Non-subscribers are free to use the similar package built for RHEL on their Oracle Linux machines. In this case, we're downloading the similar package built for RHEL (available here)
Let's complete the Installation of Oracleasm-support and oracleasmlib.

[root@dbnode01 ~]# wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el5.x86_64.rpm .................. Saving to: oracleasmlib-2.0.4-1.el5.x86_64.rpm 100%[============>] 14,176 --.-K/s in 0.01s 2014-09-30 21:53:10 (1.26 MB/s) - oracleasmlib-2.0.4-1.el5.x86_64.rpm saved [14176/14176]

[root@dbnode01 ~] yum localinstall oracleasmlib-2.0.4-1.el5.x86_64.rpm Loaded plugins: security Setting up Local Package Process .................. Installed: oracleasmlib.x86_64 0:2.0.4-1.el5 Complete!

[root@dbnode01 ~] yum install oracleasm-support Loaded plugins: security Setting up Install Process .................. Installed: oracleasm-support.x86_64 0:2.1.8-1.el6 Complete!

There are some configuration to apply after ASMLib Installation.In fact, ASMLib is unaware of the multipath configuration. It can use any of the paths, sdb, sdc, or multipatha, and it won't know the difference. The default configuration does not care which path it uses.
In its default configuration, ASMLib will choose the first path it finds. This is the first path as reported by Linux. Depending on your storage driver, it could be the multipath, or it could be one of the single paths. We want ASMLib to always use the multipath disk and that must be told via its configuration.
On RHEL like system, the main configuration file is /etc/sysconfig/oracleasm (part of oracleasm-support). The following 02 lines are added to this file, the first is instructing ASMlib to scan the multipath disks (starting with dm under /dev) and the second to ignore all sd drives.

** /etc/sysconfig/oracleasm **
ORACLEASM_SCANORDER=”dm” ORACLEASM_SCANEXCLUDE=”sd”

Note that when scanning the devices, only the device names known by the kernel are scanned. With device-mapper, the kernel sees the devices as /dev/dm-XX.

The /dev/mapper/XXX names are created by udev for human readability.So, Any configuration of ORACLEASM_SCANORDER or ORACLEASM_SCANEXCLUDE must use the dm prefix.

We can now complete the oracleasm configuration by running the oracleasm command with its initial configuration switch (-i) , Note that this initial configuration will re-write the /etc/sysconfig/oracleasm (but will keep SCANORDER and SCANEXCLUDE configuration done before). This must be performed on both nodes.

[root@dbnode01 ~]# /usr/sbin/oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <enter> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [oracle]: grid Default group to own the driver interface [oinstall]: asmdba Start Oracle ASM library driver on boot (y/n) [y]: Scan for Oracle ASM disks on boot (y/n) [y]: Writing Oracle ASM library driver configuration: done

Load ASM Module manually (on both nodes),

[root@dbnode01 oracle]# /usr/sbin/oracleasm init Creating /dev/oracleasm mount point: /dev/oracleasm Loading module "oracleasm": oracleasm Configuring "oracleasm" to use device physical block size Mounting ASMlib driver filesystem: /dev/oracleasm

Now we can create our ASM Disk, this must be done only on one node.

[root@dbnode01 ~]# cat /tmp/create_initial_asm.sh #!/bin/bash set -x /usr/sbin/oracleasm createdisk OCR_DISK1 /dev/mapper/asm-ocr-disk1p1 /usr/sbin/oracleasm createdisk OCR_DISK2 /dev/mapper/asm-ocr-disk2p1 /usr/sbin/oracleasm createdisk OCR_DISK3 /dev/mapper/asm-ocr-disk3p1 /usr/sbin/oracleasm createdisk DATA_DISK1 /dev/mapper/asm-data-disk1p1 [root@dbnode01 ~]# chmod ug+x /tmp/create_initial_asm.sh [root@dbnode01 ~]# /tmp/create_initial_asm.sh

Scan and list the newly created disks on both nodes,

[root@dbnode01 ~]# /usr/sbin/oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... [root@dbnode01 ~]# /usr/sbin/oracleasm listdisks DATA_DISK1 OCR_DISK1 OCR_DISK2 OCR_DISK3

With that done, we are done with this Section and can start the Oracle Grid Infrastructure Software Installation...

III. Oracle Grid 11GR2 Software Installation:

Let's start the grid Infrastructure Software Installation by running runInstaller (needs graphical environment) as Grid User as described in the screenshots below.

[grid@dbnode01 grid]# ./runInstaller 

At this stage, we will need to add the other cluster node, identify Network Interfaces and Setup/Test SSH Connectivity.

Click on SSH Connectivity, fill OS Password and run a Test and then setup.

Testing Again,

Click on "identify Network Interface", and mark both private/public Interfaces

With that completed, we can move to the next step where the Cluster Registery Storage Type will be set to "Oracle Automatic Storage Management" and we'll specify SYSASM password.

Just a few comments about the following  section (Disk Group Selection)

For Disk Group Name— Enter a name, for example: OCR_VOTE.

For Redundancy— For your OCR and voting disks, select High if five ASM disks are available, select Normal if three ASM disks are available, or select External if one ASM disk is available and you're managing redundancy in an external disk (not recommended).

Also, If no candidate disks are displayed, click Change Discovery Path and enter ORCL:*or /dev/oracleasm/disks/*. Ensure that you have marked your Oracle ASM disks, for more informations see,"Using ASMLib to Mark the Shared Disks as Candidate Disks"

Note that for the step below,  I faced the bug identified here ( Metalink Note 1267569.1) "Device check for ASM" and choose to ignore (as recommended).

At this stage, we should go back to the Shell and run the 02 scripts requested as root. For every script, run it on the local node first, after successful completion we can run in parallel on other nodes.

[root@dbnode01 rpm]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@dbnode02 rpm]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@dbnode01 rpm]# /u01/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding Clusterware entries to upstart CRS-2672: Attempting to start 'ora.mdnsd' on 'dbnode01' CRS-2676: Start of 'ora.mdnsd' on 'dbnode01' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'dbnode01' CRS-2676: Start of 'ora.gpnpd' on 'dbnode01' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode01' CRS-2672: Attempting to start 'ora.gipcd' on 'dbnode01' CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode01' succeeded CRS-2676: Start of 'ora.gipcd' on 'dbnode01' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'dbnode01' CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode01' CRS-2676: Start of 'ora.diskmon' on 'dbnode01' succeeded CRS-2676: Start of 'ora.cssd' on 'dbnode01' succeeded ASM created and started successfully. Disk Group OCR_DG created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 585672299e6f4f27bff2454b6bf4a69d. Successful addition of voting disk 85236f21fb224f58bf8bf56da150e2b9. Successful addition of voting disk 2186a37114854fffbfbd416ed2ddd250. Successfully replaced voting disk group with +OCR_DG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 585672299e6f4f27bff2454b6bf4a69d (ORCL:OCR_DISK1) [OCR_DG] 2. ONLINE 85236f21fb224f58bf8bf56da150e2b9 (ORCL:OCR_DISK2) [OCR_DG] 3. ONLINE 2186a37114854fffbfbd416ed2ddd250 (ORCL:OCR_DISK3) [OCR_DG] Located 3 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'dbnode01' CRS-2676: Start of 'ora.asm' on 'dbnode01' succeeded CRS-2672: Attempting to start 'ora.OCR_DG.dg' on 'dbnode01' CRS-2676: Start of 'ora.OCR_DG.dg' on 'dbnode01' succeeded Configure Oracle Grid Infrastructure for a Cluster ... succeeded [root@dbnode02 rpm]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@dbnode02 rpm]# /u01/11.2.0/grid/root root.sh rootupgrade.sh [root@dbnode02 rpm]# /u01/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful Adding Clusterware entries to upstart CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node dbnode01, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster Configure Oracle Grid Infrastructure for a Cluster ... succeeded

IV. Installing Oracle 11g R2 Database (RDBMS) Software:

As Oracle User,  let's run runInstaller

[oracle@dbnode01 database]# ./runInstaller 

Click on SSH Connectivity and Setup/Test the connectivity

Again, we've to go back to the shell as root and run the following on both nodes.

[root@dbnode01 rpm]# /u01/app/oracle/product/11.2.0/db_1/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Finished product-specific root actions.

With both Grid and Database Software Installation completed, the next step will most probably be the creation of Data Disk Group and DB Creation...

References:
http://sethmiller.org/it/oracleasmlib-not-necessary/
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/3336.how-to-deploy-oracle-11gr2-on-rhel6oracle-linux-6
http://uhesse.com/2010/10/06/voting-disk-and-ocr-in-11gr2-some-changes/
http://dnccfg.blogspot.com/2012/09/koracle11gr2racsetup.html
http://oracleinaction.com/haip/