Upgrading Oracle 11.2.0.1 RAC Clusters to 11.2.0.2 on Linux x86-64

This page discusses an issue which you may hit if attempting to upgrade an Oracle 11.2.0.1 Linux x86-64 RAC cluster to Oracle 11.2.0.2.

The Grid Infrastructure upgrade must be downloaded from MOS (Patch Number 10098816) This download consists of seven zip files; the Grid Infrastructure file is called:

p10098816_112020_Linux-x86-64_3of7zip
Unzip the file into a staging directory e.g. /home/grid/stage; a subdirectory called grid will be created. Start the upgrade using
[grid@server14] $ cd /home/grid/stage/grid
[grid@server14] $ ./runInstaller

The OUI should detect that Grid Infrastructure has already been installed. The upgrade interview is very similar to installation, though obviously there are fewer options as many decisions cannot be changed without a complete reinstallation.

When you reach the Summary page, press Finish to end the interview process and to start the install. The installer will copy files to the local node, relink them and distribute the relinked executables to the remote node(s). The installer will then prompt you to run the rootupgrade.sh script as the root user.

This is where we hit a problem. When we ran the rootupgrade.sh script on the first node it failed with the following error:

[root@server14]# /u01/app/11.2.0.2/grid/rootupgrade.sh
Running Oracle 11g root script...
The following environment variables are set as:
  ORACLE_OWNER= grid
  ORACLE_HOME= /u01/app/11.2.0.2/grid 
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed.  No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: 
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/11.2.0.2/grid/crs/install/crsconfig_params
Creating trace directory
Failed to add (property/value):('OLD_OCR_ID/'-1') for
checkpoint:ROOTCRS_OLDHOMEINFO.Error code is 256
The fixes for bug 9413827 are not present in the 11.2.0.1
crs home
Apply the patches for these bugs in the 11.2.0.1 crs home
and then run rootupgrade.sh
/u01/app/11.2.0.2/grid/perl/bin/perl
  -I/u01/app/11.2.0.2/grid/perl/lib\n
  -I/u01/app/11.2.0.2/grid/crs/install 
  /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl execution failed
Although the script is requesting fixes for 9413827, searches for this bug on MOS reveal that it is actually included in bug 9655006 which turns out to be the 11.2.0.1.2 PSU which must be downloaded from MOS. The download file is:
p9655006_112010_Linux-x86-64.zip
This zip file actually contains two patches, 9655006 and 9654983. and must be unzipped into an EMPTY directory. PSUs are normally installed using the new opatch auto option which attempts to install all patches it encounters in the specified directory - if it sees any other directories it will assume they are patches and may fail with a fatal error if they are not.

For example if the PSU has been unzipped into the directory /home/grid/PSU, then the command to install the PSU might be:

  /u01/app/11.2.0/grid/OPatch/opatch auto /home/grid/stage
   -oh /u01/app/11.2.0/grid

where the -oh parameter specifies the Oracle home for the Grid Infrastructure (in this case /u01/app/11.2.0/grid)

Prior to installing the PSU, check that you have the correct version of opatch (to include the auto option) which should be 11.2.0.1.3 or above. If you currently have Oracle 11.2.0.1, then opatch may be out of date. Check using:

[grid@server14] $cd /u01/app/11.2.0/grid/OPatch
[grid@server14]$ opatch version
Invoking OPatch 11.1.0.6.6
OPatch Version: 11.1.0.6.6
OPatch succeeded
Download opatch from MOS. We used patch 6880880 and downloaded the following file:
p6880880_112000_Linux-x86-64.zip
We downloaded the file into the Grid Infrastructure home. When you unzip this file it overwrites the files in the existing OPatch directory. After unzipping the file the opatch utillity should have been upgrading. You can verify this using:
[grid@server14] $cd /u01/app/11.2.0/grid/OPatch
[grid@server14]$ opatch version
Invoking OPatch 11.2.0.1.3
OPatch Version: 11.2.0.1.3
OPatch succeeded