Oracle RAC and ActiveCluster v01
Oracle RAC and ActiveCluster v01
http://www.purestorage.com
•
•
•
But perhaps the most important advantage of this architecture is the degree to which it simplifies
disaster recovery, both for testing and for recovering from actual disasters. If application servers
are configured to distribute the workload over the entire cluster, recovery is zero-touch (requiring
no human intervention), zero-RPO, and zero-RTO. Moreover, systems using this architecture
self-heal when failed sites come back online.
•
•
•
•
•
•
•
Although Purity ActiveCluster is quite simple to configure, this report includes configuration of
the entire solution software stack, in part to provide the rationale for certain decisions. The
example section describes storage and RAC node configuration, installation of Oracle Grid
Infrastructure, ASM configuration, Oracle RAC database software installation, and
Swingbench installation, setup, and database creation. For readers familiar with all of these
except ActiveCluster configuration, the latter consists of four steps:
chkconfig iscsid on
service iscsid start
chkconfig iscsi on
service iscsi start
/etc/iscsi/initiatorname.iscsi
[root@acrac1 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1988-12.com.oracle:a5338d66785
acrac1
acrac2
acrac1 acrac2
acrac1
acrac1
acrac1
acrac2
FlashArrays also support FlashRecover asynchronous replication, which prioritizes local write
performance over RPO. FlashRecover is suitable for less critical write-intensive systems such as
extract-transform-load (ETL)-based reporting and decision support.
With both synchronous and asynchronous replication available, users can make the right
performance-RPO tradeoff for each application’s service level requirement.
Unlike rotating disk storage, the number of IOPS a FlashArray can deliver is independent of the
number of LUNs in an ASM diskgroup. This makes it possible to significantly reduce the number
of objects managed by ASM. Ten 1-terabyte LUNs provide the same performance as a single 10-
terabyte LUN. To minimize management overhead, therefore, the example uses a single LUN per
ASM diskgroup. Moreover, because (a) FlashArray volumes (LUNs) can be resized instantly, and
(b) ASM 12c supports device resizing, there is no need to add LUNs to an ASM diskgroup to
increase its capacity or performance.
acrac1
acrac2
The following steps are specific to iSCSI arrays. Different steps would be requires for arrays that
use Fibre Channel to connect to hosts.
ETH4
192.168.150.*
[root@acrac1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-iscsi1
HWADDR=00:25:B5:A0:00:0F
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.150.25
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=iscsi1
UUID=2f75d5a4-b316-4234-aa35-2cec390d1abb
DEVICE=iscsi1
ONBOOT=yes
ifup
sdc sdg
/etc/multipath.conf
defaults {
find_multipaths yes
polling_interval 10
}
devices {
device {
vendor "PURE"
path_selector "queue-length 0"
path_grouping_policy group_by_prio
path_checker tur
fast_io_fail_tmo 10
dev_loss_tmo 60
no_path_retry 0
hardware_handler "1 alua"
prio alua
failback immediate
}
}
multipaths {
multipath {
wwid 3624a93702e81aa5e8c3c4aec00011014
alias dg_acrac_crs
}
multipath {
wwid 3624a93702e81aa5e8c3c4aec00011cbd
alias dg_acrac_mgmt
}
multipath {
wwid 3624a93702e81aa5e8c3c4aec00011013
alias dg_acrac_activedb_control_redo
}
multipath {
wwid 3624a93702e81aa5e8c3c4aec00011010
alias dg_acrac_activedb_data
}
multipath {
wwid 3624a93702e81aa5e8c3c4aec00011012
alias dg_acrac_activedb_fra
}
}
blacklist {
}
multipath.conf
multipath.conf
multipathd start
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
fdisk
fdisk /dev/mapper/dg_acrac_mgmt
fdisk /dev/mapper/dg_acrac_activedb_control_redo
fdisk /dev/mapper/dg_acrac_activedb_data
fdisk /dev/mapper/dg_acrac_activedb_fra
acrac1 acrac2
srvctl add service -db ACTIVEDB -service SWINGBENCH_AB -preferred
"ACTIVEDB1,ACTIVEDB2" -failovertype TRANSACTION -replay_init_time 300 -failoverretry
30 -failoverdelay 3 -commit_outcome TRUE -failover_restore LEVEL1 -drain_timeout 60 -
stopoption IMMEDIATE
acrac1
srvctl add service -db ACTIVEDB -service SWINGBENCH_A.puresg.com -preferred ACTIVEDB1
-available ACTIVEDB2 -failovertype TRANSACTION -replay_init_time 300 -failoverretry 30
-failoverdelay 3 -commit_outcome TRUE -failover_restore LEVEL1 -drain_timeout 60 -
stopoption IMMEDIATE
$SWINGHOME/bin/swingbench
$SWINGHOME/bin/swingbench.xml
vi swingbench.xml
•
•
•
•
•
sudo multipath -ll dg_acrac_activedb_data
acrac2
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 250
2 250
acrac1
-1 acrac2
•
•
• acrac2
But perhaps the most important result demonstrated by this test is that no administrative action
whatsoever was required during RAC node acrac2‘s switch to using FLASHARRAY-A.
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 250
2 250
sudo multipath -ll dg_acrac_activedb_data
•
•
•
•
puresgn5k# show interface eth1/19-22 brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth1/19 40 eth trunk up none 10G(D) --
Eth1/20 40 eth trunk up none 10G(D) --
Eth1/21 40 eth trunk up none 10G(D) --
Eth1/22 40 eth trunk up none 10G(D) –
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 250
2 250
puresgn5k# config t
Enter configuration commands, one per line. End with CNTL/Z.
puresgn5k(config)# interface eth1/19-22
puresgn5k(config-if-range)# shutdown
puresgn5k(config-if-range)# show interface eth1/19-22 brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth1/19 40 eth trunk down Administratively down 10G(D) --
Eth1/20 40 eth trunk down Administratively down 10G(D) --
Eth1/21 40 eth trunk down Administratively down 10G(D) --
Eth1/22 40 eth trunk down Administratively down 10G(D) --
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 250
2 250
sudo multipath -ll dg_acrac_activedb_data
acrac1
acrac2
•
•
•
•
• acrac1
puresgn5k# config t
Enter configuration commands, one per line. End with CNTL/Z.
puresgn5k(config)# interface eth1/19-22
puresgn5k(config-if-range)# no shutdown
puresgn5k(config-if-range)# show interface eth1/19-22 brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth1/19 40 eth trunk up none 10G(D) --
Eth1/20 40 eth trunk up none 10G(D) --
Eth1/21 40 eth trunk up none 10G(D) --
Eth1/22 40 eth trunk up none 10G(D) --
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 250
2 250
sudo multipath -ll dg_acrac_activedb_data
•
•
• acrac1
acrac2
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 249
2 251
acrac2 ora_pmon_ACTIVEDB2
oracle@acrac2:/home/oracle [ACTIVEDB2]
> ps -ef | grep pmon
oracle 2561 1 0 Sep06 ? 00:00:10 ora_pmon_ACTIVEDB2
grid 18643 1 0 Aug26 ? 00:00:57 asm_pmon_+ASM2
grid 20074 1 0 Aug29 ? 00:00:37 mdb_pmon_-MGMTDB
oracle@acrac2:/home/oracle [ACTIVEDB2]
> date ; kill -9 2561
Thu Sep 7 17:42:02 SGT 2017
acrac1
acrac2
acrac2 acrac1
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 500
acrac2
acrac2
acrac1
acrac2 acrac1
sudo multipath -ll dg_acrac_activedb_data
acrac1
select inst_id, count(*) from gv$session where username = 'SOE' group by inst_id;
INST_ID COUNT(*)
---------- ----------
1 499
2 1
#Setup OEL 7.x Host with pre-requisites for Oracle Installation
#Storage devices may need to be adjusted as appropriate
#To be run as root
#!/bin/bash
##
### Create Oracle user accounts and groups
##
# Groups
echo "dba::501:oracle,oinstall,oemop" >> /etc/group
echo "oinstall::502:oracle" >> /etc/group
# Users
useradd -m -u 555 -g oinstall -G dba,asmadmin,asmdba,oper -d /home/oracle -s /bin/bash
-c "Oracle Superuser" oracle
useradd -m -u 556 -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid -s
/bin/bash -c "Oracle Grid" grid
passwd -x -1 oracle
passwd -x -1 grid
# Optional Volume & Filesystem for OEM Agent (skip if OEM is not used in the
environment
lvcreate -L5g -noraagentvol ol
mkfs -t ext4 /dev/ol/oraagentvol
mkdir -p /u01/app/oracle/product/agent
chmod 775 /u01/app/oracle/product/agent
echo "/dev/ol/oraagentvol /u01/app/oracle/product/agent ext4 defaults,discard
1 2 " >> /etc/fstab
chown -R oracle:oinstall /u01/app/oracle/product/agent
mount /u01/app/oracle/product/agent
chown -R oracle:oinstall /u01/app/oracle/product/agent
# DB Binaries
lvcreate -L25g -noradbvol ol
mkfs -t ext4 /dev/ol/oradbvol
mkdir -p /u01/app/oracle/product/db
chmod 775 /u01/app/oracle/product/db
echo "/dev/ol/oradbvol /u01/app/oracle/product/db ext4 defaults,discard
1 2 " >> /etc/fstab
chown -R oracle:oinstall /u01/app/oracle/product/db
mount /u01/app/oracle/product/db
chown -R oracle:oinstall /u01/app/oracle/product/db
# Create local Volume & Filesystem for diagnostics and log files area
lvcreate -L25g -noralocalvol ol
mkfs -t ext4 /dev/ol/oralocalvol
mkdir -p /u01/app/oracle/local
chmod 775 /u01/app/oracle/local
echo "/dev/ol/oralocalvol /u01/app/oracle/local ext4 defaults,discard 1
2 " >> /etc/fstab
chown -R oracle:oinstall /u01/app/oracle/local
mount /u01/app/oracle/local
chown -R oracle:oinstall /u01/app/oracle/local
##
### Install RPM packages required by Oracle
##
umask 022
CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
ORACLE_LIST="binutils compat-libstdc++ elfutils-libelf elfutils-libelf-devel expat gcc
gcc-c++ glibc glibc-common glibc-devel glibc-headers libaio libaio-devel libgcc
libstdc++ libstdc++-devel make pdksh sysstat unixODBC unixODBC-devel oracleasm
oracleasmlib oracleasm-support cvuqdisk openssh-clients"
##
### Oracle profile
##
cat >> /home/oracle/.bash_profile <<EOF
export ORACLE_SID=
export ORACLE_HOSTNAME=
export ORACLE_BASE=/u01/app/oracle/global
export GRID_HOME=/u01/app/grid/product/12cR2
export DB_HOME=/u01/app/oracle/product/db/12cR2
export ORA_INV=/u01/app/grid/oraInventory
export ADR_HOME=\$ORACLE_BASE/diag
export ORA_ADMIN=\$ORACLE_BASE/admin
export AGENT_HOME=/u01/app/oracle/product/agent/agent13c/agent_13.1.0.0.0
export ORACLE_HOME=\$DB_HOME
export TNS_ADMIN=\$GRID_HOME/network/admin
export LDAP_ADMIN=$TNS_ADMIN
export SQLPATH=\$ORACLE_HOME/sqlscripts
export NLS_LANG='english_united kingdom.we8iso8859p1'
export NLS_DATE_FORMAT='DD Mon YYYY HH24:MI:SS'
export LD_LIBRARY_PATH=\$ORACLE_HOME/lib
export
PATH=\$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/etc:/usr/local/bin:/sbin:\$PATH:
export EDITOR=vi
stty erase ^H
umask 022
ulimit -n 65536
ulimit -u 16384
EOF
##
### Grid profile
##
cat >> /home/grid/.bash_profile <<EOF
#
# These values need setting by the DBA
#
export ORACLE_SID=+ASM1
export ORACLE_HOSTNAME=
#
#
export ORACLE_BASE=/u01/app/grid/local
export GRID_HOME=/u01/app/grid/product/12cR2
export DB_HOME=/u01/app/oracle/product/db/12cR2
export ORA_INV=/u01/app/grid/oraInventory
export ADR_HOME=\$ORACLE_BASE/diag
export ORA_ADMIN=\$ORACLE_BASE/admin
export AGENT_HOME=/u01/app/oracle/product/agent/agent13c/agent_13.1.0.0.0
export ORACLE_HOME=\$GRID_HOME
export TNS_ADMIN=\$GRID_HOME/network/admin
export LDAP_ADMIN=$TNS_ADMIN
export NLS_LANG='english_united kingdom.we8iso8859p1'
export NLS_DATE_FORMAT='DD Mon YYYY HH24:MI:SS'
export LD_LIBRARY_PATH=\$ORACLE_HOME/lib
export
PATH=\$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/etc:/usr/local/bin:/sbin:\$PATH:
export EDITOR=vi
stty erase ^H
umask 022
ulimit -n 65536
ulimit -u 16384
EOF
##
### Directories/Permissions
##
chown oracle:oinstall /home/oracle/.profile
chown grid:oinstall /home/grid/.profile
chmod 750 /home/oracle /home/grid
chmod 640 /home/oracle/.profile /home/grid/.profile
mkdir -p /u01/app/oracle/product/db
chown -R oracle:oinstall /u01/app/oracle
mkdir -p /u01/app/grid/product/
chown -R grid:oinstall /u01/app/grid
mkdir -p /u01/app/grid/oraInventory
chown grid:oinstall /u01/app/grid/oraInventory
mkdir -p /var/opt/oracle
chown -R oracle:oinstall /var/opt/oracle
ln -s /etc/oratab /var/opt/oracle/oratab
##
### Set oracle/grid ulimits in /etc/profile
##
echo 'if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
' >> /etc/profile
##
### /etc/security/limits.conf
##
echo >> /etc/security/limits.conf
echo "# Oracle Limits" >> /etc/security/limits.conf
echo "oracle soft nproc 2047" >> /etc/security/limits.conf
echo "oracle hard nproc 16384" >> /etc/security/limits.conf
echo "oracle soft nofile 4096" >> /etc/security/limits.conf
echo "oracle hard nofile 65536" >> /etc/security/limits.conf
echo "oracle soft memlock 25165824" >> /etc/security/limits.conf
echo "oracle hard memlock 25165824" >> /etc/security/limits.conf
echo "grid soft nproc 2047" >> /etc/security/limits.conf
echo "grid hard nproc 16384" >> /etc/security/limits.conf
echo "grid soft nofile 4096" >> /etc/security/limits.conf
echo "grid hard nofile 65536" >> /etc/security/limits.conf
echo "grid soft memlock 25165824" >> /etc/security/limits.conf
echo "grid hard memlock 25165824" >> /etc/security/limits.conf
##
### /etc/sysctl.conf
##
echo >> /etc/sysctl.conf
echo "# Oracle Settings" >> /etc/sysctl.conf
# pagecache
echo "vm.pagecache=30" >> /etc/sysctl.conf
# Shared Memory
echo "kernel.shmmni=4096" >> /etc/sysctl.conf
# The default build sets these high enough
# echo "kernel.shmall=17039360" >> /etc/sysctl.conf
# echo "kernel.shmmax=69793218560" >> /etc/sysctl.conf
# kernel.sem
echo "kernel.sem=268 32000 100 256" >> /etc/sysctl.conf
# Disable ASLR
echo "kernel.randomize_va_space=0" >> /etc/sysctl.conf
echo "kernel.exec-shield=0" >> /etc/sysctl.conf
# fs
echo "fs.file-max=6815744" >> /etc/sysctl.conf
echo "fs.aio-max-nr=1048576" >> /etc/sysctl.conf
# Aggressive swapping
echo "vm.swappiness=100" >> /etc/sysctl.conf
# net
echo "net.core.rmem_default=262144" >> /etc/sysctl.conf
echo "net.core.wmem_default=262144" >> /etc/sysctl.conf
echo "net.core.rmem_max=4194304" >> /etc/sysctl.conf
echo "net.core.wmem_max=1048576" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range=9000 65500" >> /etc/sysctl.conf
##
### Pam
##
echo "session required pam_limits.so" >> /etc/pam.d/login
echo "session required pam_limits.so" >> /etc/pam.d/sshd
##
### SUDO
##
echo "oracle ALL=(root) NOPASSWD: ALL" >> /etc/sudoers
echo "grid ALL=(root) NOPASSWD: ALL" >> /etc/sudoers
echo 'Defaults:oracle !requiretty' >> /etc/sudo.d/sudoers_local.inc
##
### Disable tty requirment from ssh sudo sessions for Oracle EM provisioning
##
chmod u+w /etc/sudoers
ex /etc/sudoers << EOF > /dev/null 2>&1
g/Defaults requiretty/s//#Defaults requiretty/g
wq
EOF
chmod u-w /etc/sudoers
cd $GRID_HOME
gridSetup.sh
gridSetup
acrac2.puresg.com
/dev/mapper
Note: In Dialog B.17 There is a bug on NTP (PRVG-13602) confirmed by Oracle support with bug
number (24314803) so ignore any NTP errors on these checks
asmca
runInstaller
###########################
admin@array-3-ct0:~$ purehost list
# SWINGBENCH CONFIG
export JAVAHOME=/u01/app/oracle/local/benchmark_tools/jdk1.8.0_141
export SWINGHOME=/u01/app/oracle/local/benchmark_tools/swingbench
export ANTHOME=$SWINGHOME/lib
export
LD_LIBRARY_PATH=/u01/app/oracle/local/benchmark_tools/instantclient_12_2:${LD_LIBRARY_
PATH}
export
CLASSPATH=$JAVAHOME/lib/rt.jar:$JAVAHOME/lib/tools.jar:$ORACLE_HOME/jdbc/lib/ojdbc14.j
ar:$SWINGHOME/lib/mytransactions.jar:${SWINGHOME}/lib/swingbench.jar:$ANTHOME/ant.jar
# END OF SWINGBENCH CONFIG
##########################
$SWINGHOME/bin/oewizard
oewizard
•
•
•
•
http://www.purestorage.com/blog/author/dannyhiggins