Step By Step: Upgrade 10gR2 RAC to 11gR1 RAC on
Oracle Enterprise Linux 5 (32 bit) Platform.
By Bhavin Hingu
CLICK HERE for the”Step By Step: Upgrade 10gR2 RAC to 11gR2 RAC on Linux”
This Document shows the step by step of upgrading 3-Node
10gR2 RAC database on ASM to 11gR1 RAC. The below upgrade path allows upgrading
10gR2 CRS to 11gR1 CRS with no downtime using rolling upgrade. The rolling
upgrade of ASM is possible starting from Release 11gR1 (11.1.0.6) and is not
backward compatible. That means we cannot use Rolling upgrade method for
upgrading 10gR2 ASM to 11gR1 ASM.
Upgrading 10gR2 RAC database will need outage and the
total downtime may further be avoided or minimized by using Logical Standby
Database in the upgrade process (not covered here).
|
Existing 10gR2 RAC setup (Before Upgrade) |
Target 11gR1 RAC Setup (After Upgrade) |
Clusterware |
Oracle 10g R2 Clusterware 10.2.0.3 |
Oracle 11gR1 Clusterware 11.1.0.6 |
ASM
Binaries |
10g R2 RAC 10.2.0.3 |
11gR1 RAC 11.1.0.6 |
Cluster
Name |
Lab |
Lab |
Cluster
Nodes |
node1, node2, node3 |
node1, node2, node3 |
Clusterware
Home |
/u01/app/oracle/crs (CRS_HOME) |
/u01/app/oracle/crs (CRS_HOME) |
Clusterware
Owner |
oracle:(oinstall, dba) |
oracle:(oinstall, dba) |
VIPs |
node1-vip, node2-vip, node3-vip |
node1-vip, node2-vip, node3-vip |
SCAN |
N/A |
N/A |
SCAN_LISTENER
Host/port |
N/A |
N/A |
OCR
and Voting Disks Storage Type |
Raw Devices |
Raw Devices |
OCR
Disks |
/dev/raw/raw1, /dev/raw/raw2 |
/dev/raw/raw1, /dev/raw/raw2 |
Voting
Disks |
/dev/raw/raw3, /dev/raw/raw4,
/dev/raw/raw5 |
/dev/raw/raw3, /dev/raw/raw4,
/dev/raw/raw5 |
ASM_HOME |
/u01/app/oracle/asm |
/u01/app/oracle/asm11gr1 |
ASM_HOME
Owner |
oracle:(oinstall, dba) |
oracle:(oinstall, dba) |
ASMLib
user:group |
oracle:oinstall |
oracle:oinstall |
|
||
DB
Binaries |
Oracle 10gR2 RAC (10.2.0.3) |
Oracle 11gR1 RAC (11.1.0.6) |
DB_HOME |
/u01/app/oracle/db |
/u01/app/oracle/db11gr1 |
DB_HOME
Owner |
oracle:(oinstall, dba) |
oracle:(oinstall, dba) |
DB LISTENER |
LAB_LISTENER |
LAB_LISTENER |
DB
Listener Host/port |
node1-vip, node2-vip, node3-vip
(port 1530) |
node1-vip, node2-vip, node3-vip
(port 1530) |
DB
Storage Type, File Management |
ASM with OMFs |
ASM with OMFs |
ASM
diskgroups for DB and FRA |
DATA, FRA |
DATA, FRA |
OS
Platform |
Oracle Enterprise Linux 5.5 (32 bit) |
Oracle Enterprise Linux 5.5 (32 bit) |
HERE’s the Existing 10gR2 RAC Setup in Detail
The Upgrade Process is composed of below 4 Stages:
·
Pre-Upgrade
Tasks
·
Upgrade 10gR2 (10.2.0.1) Clusterware to 11gR1 (11.1.0.6) ß Rolling Upgrade No
Downtime.
·
Upgrade 10gR2 ASM_HOME (10.2.0.1) to 11gR1 RAC (11.1.0.6) ß ASM and Database
Downtime required.
·
Upgrade 10gR2 Database to 11gR1 RAC (11.1.0.6) ß Database Downtime
required.
Pre-Upgrade tasks:
Install/Upgrade RPMs
required for 11gR1 RAC and 11gR2 RAC Installation
Set 11gR1 specific
Kernel Parameters
Update the TIMEZONE file
version
Backing up
ORACLE_HOMEs/database
Minimum Required RPMs for 11gR1 RAC on OEL 5.5
(All the 3 RAC Nodes):
binutils-2.17.50.0.6-2.el5
compat-libstdc++-33-3.2.3-61
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
glibc-2.5-12
glibc-common-2.5-12
glibc-devel-2.5-12
glibc-headers-2.3.4-2
gcc-4.1.1-52
gcc-c++-4.1.1-52
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.1-52
libstdc++-4.1.1
libstdc++-devel-4.1.1-52.e15
make-3.81-1.1
sysstat-7.0.0
unixODBC-2.2.11
unixODBC-devel-2.2.11
Below
command verifies whether the specified rpms are installed or not. Any missing
rpms can be installed from the OEL Media Pack
rpm -q
binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel \
gcc
gcc-c++ glibc glibc-common glibc-devel glibc-headers libaio libaio-devel \
libgcc
libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel
NOTE: The cvuqdisk rpm in 11gR1 has the same version as one available in
10gR2 (cvuqdisk-1.0.1-1), so I did not have to replace the 10gR2 cvuqdisk by
11gR1 one.
Kernel Parameter Requirements (All the 3 RAC Nodes):
/etc/sysctl.conf:
# Kernel sysctl configuration file for Oracle
Enterprise Linux
#
# For binary values, 0 is
disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more
details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source
routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging
functionality of the kernel
kernel.sysrq = 1
# Controls whether core dumps will append the PID to
the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage
queue
kernel.msgmax = 8192
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295
# Controls the maximum number of shared memory segments,
in pages
kernel.shmall = 1073741824
# For 11g, recommended
value for file-max is 6815744
fs.file-max = 6815744
# For 10g, uncomment
'fs.file-max 327679', comment other entries for this parameter and re-run
sysctl -p
# fs.file-max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
# For 11g, recommended
value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment
'net.core.rmem_max 2097152', comment other entries for this parameter and
re-run sysctl -p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended
value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment
'net.core.wmem_max 262144', comment other entries for this parameter and re-run
sysctl -p
# net.core.wmem_max:262144
fs.aio-max-nr = 3145728
# For 11g, recommended
value for ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
# For 10g, uncomment
'net.ipv4.ip_local_port_range 1024 65000', comment other entries for this
parameter and re-run sysctl -p
# net.ipv4.ip_local_port_range:1024 65000
# Added min_free_kbytes
50MB to avoid OOM killer on EL4/EL5
vm.min_free_kbytes = 51200
Enable
these changes in the kernel.
sysctl -p
Update
the TIMEZONE file Version:
The below
database patches were applied in order to update the timezone file version to 4
if it is less than 4. DBUA may fail with the “ORA-01722: invalid number” error during the upgrade process.
Verify the current version of the timezone file
in the 10gR2 database.
select * from
v$timezone_file;
Patches to update the timezone file to version
4:
5746875
5632264
Backing Up ORACLE_HOMEs/database:
Steps I followed to take the Backup of ORACLE_HOMEs before
the upgrade: (This can be applied to 11gR1 and 10g databases)
On node1:
mkdir backup
cd backup
dd
if=/dev/dev/raw1 of=ocr_disk_10gr2.bkp
dd
if=/dev/dev/raw3
of=voting_disk_10gr2.bkp
tar cvf
node1_crs_10gr2.tar /u01/app/oracle/crs/*
tar cvf
node1_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf
node1_db_10gr2.tar /u01/app/oracle/db/*
tar cvf
node1_etc_oracle /etc/oracle/*
cp
/etc/inittab etc_inittab
mkdir etc_init_d
cd etc_init_d
cp
/etc/init.d/init* .
On node2:
mkdir backup
cd backup
tar cvf
node2_crs_10gr2.tar /u01/app/oracle/crs/*
tar cvf
node2_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf
node2_db_10gr2.tar /u01/app/oracle/db/*
tar cvf
node2_etc_oracle /etc/oracle/*
cp
/etc/inittab etc_inittab
mkdir etc_init_d
cd etc_init_d
cp /etc/init.d/init*
.
On node3:
mkdir backup
cd backup
tar cvf
node3_crs_10gr2.tar /u01/app/oracle/crs/*
tar cvf
node3_asm_10gr2.tar /u01/app/oracle/asm/*
tar cvf
node3_db_10gr2.tar /u01/app/oracle/db/*
tar cvf
node3_etc_oracle /etc/oracle/*
cp
/etc/inittab etc_inittab
mkdir etc_init_d
cd etc_init_d
cp
/etc/init.d/init* .
RMAN full
backup was taken.
Step By Step: Upgrade Clusterware, ASM and Database from
10.2.0.3 to 11.1.0.6.
Upgrade 10gR2 CRS to 11gR1 CRS in Rolling Fashion (No Downtime):
On Node1:
·
Stop the labdb1 instance, asm instance, nodeapps.
·
Relocate any db or cluster service resources to the
other nodes.
·
Stop and disable the CRS on this node
·
Reboot the node to clean up any oracle/CRS
related processes on this node.
·
Enable the CRS after the node comes back up.
·
Run the preupdate.sh script from 11gR1 clusterware Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
·
Start the runInstaller from the 11gR1 Clusterware
software directory.
srvctl
stop instance -d labdb –i labdb1
srvctl
stop asm –n node1
srvctl
stop nodeapps –n node1
crs_relocate ora.labdb.db
crs_relocate ora.labdb.oltp.cs
crsctl
stop crs
crsctl
disable crs
reboot
crsctl
enable crs ß
once the node comes back up after the reboot.
Crsctl check crs ß
CRS stack will not be up as CRS was disabled before the reboot.
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh
–crshome /u01/app/oracle/crs –crsuser oracle -noshutdown
./runInstaller
–ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade ß
run this script at the end as root
Verify the CRS
crsctl
check crs (on all the RAC nodes) ß
it should be up on all the nodes
crsctl
query crs activeversion node1 ß
The active version should be 10.2.0.3
crsctl
query crs activeversion node2
crsctl
query crs activeversion node3
crsctl
query crs softwareversion node1 ß
The Software Version should show 11.1.0.6
crsctl
query crs softwareversion node2
crsctl
query crs softwareversion node3
On Node2:
·
Stop the database instance labdb2, asm instance, nodeapps.
·
Relocate any db or cluster service resources to the
other nodes.
·
Stop and disable the CRS on this node
·
Reboot the node to clean up any oracle related
processes on this node.
·
Enable the crs after the node comes back up.
·
Run the preupdate.sh script from 11gR1 clusterware
Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
·
Start the runInstaller from the 11gR1 Clusterware
software directory.
srvctl
stop instance -d labdb –i labdb2
srvctl
stop asm –n node2
srvctl
stop nodeapps –n node2
crs_relocate ora.labdb.db
crsctl
stop crs
crsctl
disable crs
reboot
crsctl
enable crs
crsctl
check crs
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh
–crshome /u01/app/oracle/crs –crsuser oracle -noshutdown
./runInstaller
–ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade
Verify the CRS
crsctl
check crs (on all the RAC nodes)
crsctl
query crs activeversion node1 ß
The active version should be 10.2.0.3
crsctl
query crs activeversion node2 ß
The active version should be 10.2.0.3
crsctl
query crs activeversion node3
crsctl
query crs softwareversion node1 ß
The Software Version should show 11.1.0.6
crsctl
query crs softwareversion node2 ß
The Software Version should show 11.1.0.6
crsctl
query crs softwareversion node3
On Node3:
·
Stop the database instance labdb3, asm instance, nodeapps.
·
Relocate any db or cluster service resources to the
other nodes.
·
Stop and disable the CRS on this node
·
Reboot the node to clean up any oracle related
processes on this node.
·
Enable the crs after the node comes back up.
·
Run the preupdate.sh script from 11gR1 clusterware
Media to unlock the CRS_HOME (so that oracle user can modify some of the root owned files).
·
Start the runInstaller from the 11gR1 Clusterware
software directory.
srvctl
stop instance -d labdb –i labdb3
srvctl
stop asm –n node3
srvctl
stop nodeapps –n node3
crs_relocate
ora.labdb.oltp.cs
crsctl
stop crs
crsctl
disable crs
reboot
crsctl
enable crs
crsctl
check crs
/home/oracle/11gR1/clusterware/upgrade/preupdate.sh
–crshome /u01/app/oracle/crs –crsuser oracle -noshutdown
./runInstaller
–ignoreSysPrereqs
/u01/app/oracle/crs/install/rootupgrade
Verify the CRS
crsctl
check crs (on all the RAC nodes)
crsctl
query crs activeversion node1
crsctl
query crs activeversion node2
crsctl
query crs activeversion node3
crsctl
query crs softwareversion node1
crsctl
query crs softwareversion node2
crsctl
query crs softwareversion node3
The active Version and Software
version should show 11.1.0.6 at this stage.
HERE’s the detailed Screen Shots of Upgrading 10gR2 CRS to
11gR1 CRS
Upgrade 10gR2 ASM to 11gR1 ASM:
·
Start the runInstaller from the 11gR1 database
software Install Media and select the “Install Software Only” Configuration Option on the OUI screen.
./runInstaller
–ignoreSysPrereqs ß
From the 11gR1 Software for Database
Run the
root.sh on all the RAC nodes to finish the Installation.
Invoke the
dbua from the new 11gR1 ASM_HOME to upgrade the 10gR2 ASM to 11gR1 ASM.
Upgrading ASM instance is pretty much like Starting the ASM Instances and ASM
Listeners using the newly Installed 11gR1 ASM_HOME. DBUA simply copies the asm pfile and password
files from 10gR2 dbs folder to 11gR1 dbs folder. It also copies the tnsnames.ora
and listener.ora files as well. At the end, it modifies the ASM related CRS
resources (asm and all the listeners belonging to 10gR2 ASM_HOME) to start from
new 11gR1 ASM_HOME by modifying the ACTION_SCRIPT parameter to point it to the
11gR1 racgwrap.
export
ORACLE_HOME=/u01/app/oracle/asm11gr1
export
ORACLE_SID=+ASM1
/u01/app/oracle/asm11gr1/bin/dbua
HERE’s the detailed
Screen Shots of Upgrading 10gR2 ASM to 11gR1 ASM.
Issues
Encountered and Troubleshooting:
Issue #1:
During the
ASM Upgrade by DBUA, I was prompted for migrating DB listeners running from
10gR2 existing Database HOME to 11gR1 ASM HOME. I did not want the DB Listener
to migrate to ASM HOME but the Installer was not moving ahead if pressed “No”.
So, I decided to move forward with the DB listener migration. I had to manually
migrate this DB listener back to 10gR2 DB home at the
end of the upgrade of ASM process.
Here’s how
the DB listener LAB_LISTENER was moved back to 10gR2 DB HOME from 11gR1
ASM_HOME.
I made sure
that the 10gR2 TNS_ADMIN still have the listener.ora and tnsnames.ora files
with information about LAB_LISTENER.
Updated the
CRS resources related to LAB_LISTENER as shown below to point the ACTION_SCRIPT
to /u01/app/oracle/asm/bin/racgwrap
Generate the CAP files:
cd
/u01/app/oracle/crs/crs/public
/u01/app/oracle/crs/bin/crs_stat
–p ora.node1.LAB_LISTENER_NODE1.lsnr > ora.node1.LAB_LISTENER_NODE1.lsnr.cap
/u01/app/oracle/crs/bin/crs_stat
–p ora.node2.LAB_LISTENER_NODE2.lsnr > ora.node2.LAB_LISTENER_NODE2.lsnr.cap
/u01/app/oracle/crs/bin/crs_stat
–p ora.node3.LAB_LISTENER_NODE3.lsnr > ora.node3.LAB_LISTENER_NODE3.lsnr.cap
The ACTION_SCRIPT parameter was
modified in these cap files.
Replaced the line…
ACTION_SCRIPT=/u01/app/oracle/asm11gr1/bin/racgwrap
WITH…
ACTION_SCRIPT=/u01/app/oracle/asm/bin/racgwrap
Ran the below commands to update
the OCR with these changes.
/u01/app/oracle/crs/bin/crs_register
–u ora.node1.LAB_LISTENER_NODE1.lsnr
/u01/app/oracle/crs/bin/crs_register
–u ora.node2.LAB_LISTENER_NODE2.lsnr
/u01/app/oracle/crs/bin/crs_register
–u ora.node3.LAB_LISTENER_NODE3.lsnr
Issue #2:
The DBUA
could not start the ASM on node 2 during post upgrade steps at the end
of the upgrade process and so it terminated with the below error in trace.log
file.
I tried to
manually start the ASM instance and it worked just fine but the OCR was still
showing the “UNKNOWN” status for this resource. Even using crs_start/crs_stop to start or stop the ASM on node2 could not change the
status from UNKNOWN. This means that the ASM resource entry for node2 in OCR
was somehow logically corrupted by DBUA. I had to unregister this entry and
register back with the same parameter to fix the issue. After doing that, the
ASM was successfully started and stopped by srvctl command.
Generate the CAP file for ASM
resource on node2.
ora.node2.ASM2.asm.cap:
NAME=ora.node2.ASM2.asm
TYPE=application
ACTION_SCRIPT=/u01/app/oracle/asm11gr1/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=300
DESCRIPTION=CRS
application for ASM instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=node2
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=900
STOP_TIMEOUT=180
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/
as sysasm
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=
Unregister the ASM instance
resource from OCR for node2
srvctl
remove instance –d labdb –i labdb2
crs_unregister ora.node2.ASM2.asm
Registered the same ASM resource
back to OCR
crs_register ora.node2.ASM2.asm
srvctl
add instance –d labdb –i labdb2 –n node2
srvctl
modify instance –d labdb –I labdb2 –t +ASM2
srvctl
modify service -d labdb -s oltp -n -i "labdb1,labdb2,labdb3"
Upgrade
10gR2 database to 11gR1 Database:
·
Start the runInstaller from the 11gR1 database
software Install Media and select the “Install Software Only” Configuration Option on the OUI screen.
·
Invoke the dbua to upgrade the database.
·
There may be warnings about INVALID objects and
Stale Statistics which can safely be ignored.
./runInstaller
–ignoreSysPrereqs ß
From the 11gR1 Software for Database
Invoke the
DBUA from the newly installed 11gR1 HOME for the database using X terminal
/u01/app/oracle/db11gr1/bin/dbua
DBUA
invalidates some of the objects in the database during the upgrade of database
and so DBUA runs the utlrcmp.sql script at the end as part of post upgrade
steps to make them valid. I would start running this script when the DBUA
progress is around 75% so that when the DBUA runs this script to recompile the
INVALID objects, it has to recompile fewer objects and helps reduce the overall
timing of upgrade process as a whole.
Move the
listener LAB_LISTENER from 10gR2 DB home to 11gR1 HOME.
The DBUA
did not move the DB listener LAB_LISTENER from 10gR2 HOME to the 11gR1 Home and
I had to manually moved it as below:
Copy the 10gR2 listener.ora and
tnsnames.ora files to the 11gR2 TNS_ADMIN
ssh
node1 cp /u01/app/oracle/db/network/admin/listener.ora
/u01/app/oracle/db11gr1/network/admin
ssh
node1 cp /u01/app/oracle/db/network/admin/tnsnames.ora
/u01/app/oracle/db11gr1/network/admin
ssh
node2 cp /u01/app/oracle/db/network/admin/listener.ora
/u01/app/oracle/db11gr1/network/admin
ssh
node2 cp /u01/app/oracle/db/network/admin/tnsnames.ora
/u01/app/oracle/db11gr1/network/admin
ssh
node3 cp /u01/app/oracle/db/network/admin/listener.ora
/u01/app/oracle/db11gr1/network/admin
ssh
node3 cp /u01/app/oracle/db/network/admin/tnsnames.ora
/u01/app/oracle/db11gr1/network/admin
Generate the CAP files
cd
/u01/app/oracle/crs/crs/public
crs_stat -p
ora.node1.LAB_LISTENER_NODE1.lsnr > ora.node1.LAB_LISTENER_NODE1.lsnr.cap
crs_stat -p
ora.node2.LAB_LISTENER_NODE2.lsnr > ora.node2.LAB_LISTENER_NODE2.lsnr.cap
crs_stat -p
ora.node3.LAB_LISTENER_NODE3.lsnr > ora.node3.LAB_LISTENER_NODE3.lsnr.cap
Modify the ACTION_SCRIPT
parameter in these CAP files to point to the new 11gR1 home.
Update the OCR with these changes
crs_register -u
ora.node1.LAB_LISTENER_NODE1.lsnr
crs_register -u
ora.node2.LAB_LISTENER_NODE2.lsnr
crs_register -u
ora.node3.LAB_LISTENER_NODE3.lsnr
Start the listener using srvctl
srvctl
stop lsnr -l LAB_LISTENER_NODE1 -n node1
srvctl
stop lsnr -l LAB_LISTENER_NODE2 -n node2
srvctl
stop lsnr -l LAB_LISTENER_NODE3 -n node3
srvctl
start lsnr -l LAB_LISTENER_NODE1 -n node1
srvctl
start lsnr -l LAB_LISTENER_NODE2 -n node2
srvctl
start lsnr -l LAB_LISTENER_NODE3 -n node3
HERE’s the detailed
Screen Shots of Upgrading database from 10gR2 RAC to 11gR1 RAC.