Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S.
Johal
ORACLE 11G RAC: ADD SECOND NODE TO RAC ENVIRONMENT
Inderpal S. Johal
INTRODUCTION
This part of the paper will explain as how you can second Node to an existing RAC environment and it will include adding
Clusterware, RAC software as well as database instances.
STEPS FOR ADDING 2ND NODE TO RAC ENVIRONMENT
This will involves the following steps
1. Install and Configure OS and hardware for the new node
2. Add Oracle Clusterware to the new node
3. Configure ONS for the new node
4. Add RAC/Oracle Home to the new node
5. Add a Listener to the new node
6. Add a database instance to the new node
7. Troubleshooting
INSTALL AND CONFIGURE OS AND HARDWARE ON NEW NODE
Follow the steps defined in Oracle 11g RAC : Pre-requisite verifications
[db02pn]$ $ORA_CRS_HOME/bin/cluvfy stage -pre crsinst -n db02pn,db01pn -verbose -r 11gR1
Performing pre-
pre-checks for cluster services setup
Checking node reachability...
…
…
Pre-
Pre-check for cluster services setup was successful.
ADD ORACLE CLUSTERWARE TO THE NEW NODE
Step 1: Log in as the oracle user and execute the addNode.sh script located in your Oracle Clusterware home
directory on the first node. This script runs the Oracle Universal Installer.
db02pn> cd $ORA_CRS_HOME/oui/bin
db02pn> ./addNode.sh
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 10836 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 256 Passed
Oracle Universal Installer, Version 11.1.0.6.0 Production
Copyright (C) 1999, 2007, Oracle. All rights reserved.
Page 1 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 2: On the Welcome screen and Click Next to Continue
Page 2 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 3: Add Public/Private/Virtual Host name information and Next to Continue
Page 3 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 4: Click Install to Continue
Page 4 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 5: This will start the installation process
Page 5 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 6: Run the following as per the Nodes information & click Ok to Continue
Page 6 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
You will see the following error which means that you cannot delete an instance from the node where
you initiated the dbca. So we will select Node1 now in the next screen to delete all instance from Node 1
now
[root@db01pn]# /export/home/oracle/oraInventory/orainstRoot.sh
Changing permissions of /export/home/oracle/oraInventory to 770.
Changing groupname of /export/home/oracle/oraInventory to oinstall.
The execution of the script is complete
[root@db02pn ~]# /home/oracle/crs/product/11.1/crs_1/install/rootaddnode.sh
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 2: db01pn db01-priv db01pn
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/home/oracle/crs/product/11.1/crs_1/bin/srvctl add nodeapps -n db01pn -A db01-vip/255.255.255.0/eth0
[root@db01pn]# /home/oracle/crs/product/11.1/crs_1/root.sh
WARNING: directory '/home/oracle/crs/product/11.1' is not owned by root
WARNING: directory '/home/oracle/crs/product' is not owned by root
WARNING: directory '/home/oracle/crs' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS = /dev/raw/raw1
OCR backup directory '/home/oracle/crs/product/11.1/crs_1/cdata/11gcluster' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/home/oracle/crs/product/11.1' is not owned by root. Changing owner to root
The directory '/home/oracle/crs/product' is not owned by root. Changing owner to root
The directory '/home/oracle/crs' is not owned by root. Changing owner to root
The directory '/home/oracle' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: db02pn db02-priv db02pn
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
db02pn
db01pn
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M).
Page 7 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 7: Click Exit to Complete the Installation
Page 8 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
[db02pn]$ $ORA_CRS_HOME/bin/cluvfy stage -post crsinst -n db01pn
Performing post-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "db02pn".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking Cluster manager integrity...
Checking CSS daemon...
Daemon status check passed for "CSS daemon".
Cluster manager integrity check passed.
Checking cluster integrity...
Cluster integrity check passed
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check passed.
Checking CRS integrity...
Checking daemon liveness...
Liveness check passed for "CRS daemon".
Checking daemon liveness...
Liveness check passed for "CSS daemon".
Checking daemon liveness...
Liveness check passed for "EVM daemon".
Checking CRS health...
CRS health check passed.
CRS integrity check passed.
Checking node application existence...
Checking existence of VIP node application (required)
Check passed.
Checking existence of ONS node application (optional)
Check passed.
Checking existence of GSD node application (optional)
Check passed.
Post-check for cluster services setup was successful.
[db02pn]$ $ORA_CRS_HOME/bin/cluvfy stage -post crsinst -n db01pn,db02pn -verbose
Performing post-checks for cluster services setup
Checking node reachability...
…
Post-check for cluster services setup was successful
Page 9 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
CONFIGURE THE NEWNODE ONS CONFIGURATION TO OCR
You now need to add the new node ONS (Oracle Notification Server) configuration information to the shared ONS configuration
information stored in OCR. From the OLD node like GCRDB1 in our case, and looking at the ons.config file located in the <Oracle
Clusterware home>/opmn/conf directory, you can determine the ONS remote port to be used (6200 in the slide). You need to use
this port in the racgons add_config command as shown below to make sure that the ONS on the first node can communicate with
the ONS on the new node.
[db02pn] $ cat $ORA_CRS_HOME/opmn/conf/ons.config
localport=6100
remoteport=6200
loglevel=3
usesharedinstall=true
useocr=on
[db02pn] $ $ORA_CRS_HOME/bin/racgons add_config db01pn:6200
Page 10 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
ADD RAC OR ORACLE HOME TO THE NEW NODE
Step 1: Log in as the oracle user and execute the addNode.sh script located in your Oracle Clusterware home directory on
the first node. This script runs the Oracle Universal Installer.
db02pn>cd $ORACLE_HOME/oui/bin
db02pn>./addNode.sh
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 10836 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 256 Passed
Oracle Universal Installer, Version 11.1.0.6.0 Production
Copyright (C) 1999, 2007, Oracle. All rights reserved.
Page 11 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 2: On the Welcome screen and Click Next to Continue
Page 12 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 3: Select the Node 1 and Click Next to Continue
Page 13 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 4: Click Install to Continue
Page 14 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 5: This will start the installation process
Page 15 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 6: Run the following as per the Nodes information & click Ok to Continue
[root@db01pn ~]# /home/oracle/app/product/11.1/db_1/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /home/oracle/app/product/11.1/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Page 16 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 7: Click Exit to Complete the Installation
Page 17 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
ADD A DATABASE INSTANCE TO THE NEW NODE
Few Important Points
o Make sure to have all Symbolic link for Database/Logfile/Controlfile Location are available
on Node2 as it is defined in Node1. e.g. rworaclelogs etc
o Make sure that your Log group in the NODE1 are 1,2,3 and not 4,5,6 as dbca will create 4,5,6
on Node 2 and if it already exists, it will not work.
Step 1: Start the DBCA from db02pn to add Database instance to db01pn
Page 18 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 2: On the Welcome Screen Click NEXT to Continue
Page 19 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 3: Select Instance Management and Click NEXT to Continue
Page 20 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 4: Select Add an Instance and Click NEXT to Continue
Page 21 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 5: select test11g and write sys password and Click NEXT to Continue
Page 22 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 6: This will show the existing Node where instance is running and Click NEXT to Continue
Page 23 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 7: Click NEXT to Continue
Page 24 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 8: Click Finish to Continue
Page 25 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 9: Click OK to Continue
Page 26 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Page 27 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 10: Click YES to create ASM instance as well on the Node 2
Page 28 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 11: Adding Instance to the second Node
Page 29 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 12: Click YES to add LISTENER to New Node
Page 30 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 13: Continuing the Instance management
Page 31 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 14: Click OK to continue and follow the Workaround below.
Workaround
Stop the Database test11g and restart it using srvctl
[db02pn]$ srvctl stop database –d test11g
[db02pn]$ srvctl start database –d test11g
Page 32 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 15: Click NO to finish.
Page 33 8/10/2007
Oracle 11g RAC on Linux- Add Node www.datasoftech.com Inderpal S. Johal
Step 16: Validate the new RAC environment
This finishes the 2nd Node Deployment and now you can create services as per your requirement
[db01pn]$ $ORA_CRS_HOME/bin/olsnodes -p
db02pn db02-priv
db01pn db01-priv
[db01pn]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------------------------------------
ora....SM2.asm application ONLINE ONLINE db01pn
ora....PN.lsnr application ONLINE ONLINE db01pn
ora....1pn.gsd application ONLINE ONLINE db01pn
ora....1pn.ons application ONLINE ONLINE db01pn
ora....1pn.vip application ONLINE ONLINE db01pn
ora....g2.inst application ONLINE ONLINE db01pn
ora....SM1.asm application ONLINE ONLINE db02pn
ora....PN.lsnr application ONLINE ONLINE db02pn
ora....2pn.gsd application ONLINE ONLINE db02pn
ora....2pn.ons application ONLINE ONLINE db02pn
ora....2pn.vip application ONLINE ONLINE db02pn
ora.test11g.db application ONLINE ONLINE db02pn
ora....g1.inst application ONLINE ONLINE db02pn
Page 34 8/10/2007
