Here I am Patching the Grid Infrastructure & RAC home on a 2 node cluster. This cluster is on a test environment on PC.

 

This is a zero downtime patching (ZDT). Zero Downtime patching provides a process and mechanism for rolling out a patch across a domain while allowing applications to continue to service requests.

 

You can apply a ZDT patch using OPatchAuto, which rolls out the change to one node at a time and allows a load balancer   to redirect incoming traffic to the remaining nodes until the change is complete.

The Opatch utility has automated the patch application for the Oracle Grid Infrastructure (GI) home and  the Oracle RAC database homes. It operates by querying existing configurations and automating the steps required for patching each Oracle RAC database home of same version and the GI home.


The utility must be executed by an operating system (OS) user with root privileges, and it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in nonshared storage.
The utility should not be run in parallel on the cluster nodes.


Depending on command line options specified, one invocation of opatchauto can patch the GI home, Oracle RAC database homes, or both GI and Oracle RAC database homes of the same Oracle release version as the patch. You can also roll back the patch with the same selectivity.

 

The current node and the DBs on it will be in the open state. The opatchauto utility will take them down automatically while patching .

 

The version of the grid cluster and database is 12.1.0.1 on Red Hat Linux 6.4 x86_64.

 

Cluster environment is IND1202  having nodes IND120201 and IND120202.

 

Download  the below patches :

 

Latest Opatch utility.

p6880880_121010_Linux-x86-64.zip

 

Patch 23273935 Oracle Grid Infrastructure Patch Set Update 12.1.0.1.160719 (Jul2016)

p23273935_121010_Linux-x86-64.zip

 

[oracle@IND120201 patches]$ ls -ltr

total 434367

  1. drwxrwxrwx. 1 root root      4096 Jul  1 13:02 OPatch

-rwxrwxrwx. 1 root root     79747 Jul 19 19:30 PatchSearch.xml

-rwxrwxrwx. 1 root root 809453169 Aug 7 14:34 p23273935_121010_Linux-x86-64.zip

-rwxrwxrwx. 1 root root  80041425 Aug  7 20:00 p6880880_121010_Linux-x86-64.zip

  1. drwxrwxrwx. 1 root root      4096 Aug  7 20:17 23273935

[oracle@IND120201 patches]$ pwd

/mnt/hgfs/Shared_Folder/12.1.0.1/patches

 

--------------------------------------------------------------------------------------------------------------------------

OPatch Utility Information

 

You must use the OPatch utility version 12.1.0.1.7 or later to apply this patch. Oracle recommends that you use the latest released OPatch version for 12.1 releases, which is available for download from My Oracle Support patch 6880880 by selecting ARU link for the 12.1.0.1.0 release. It is recommended that you download the Opatch utility and the patch in ashared location to be able to access them from any node in the cluster for the patch application on each node.

 

When patching the GI Home, a shared location on ACFS only needs to be unmounted on the node where the GI Home is being patched.

 

The new opatch utility should be updated in all the Oracle RAC database homes and the GI home that are being patched.

 

To update Opatch, use the following instructions.


 

  1. 1. Download the OPatch utility to a temporary directory.
  2. 2. For each Oracle RAC database home and the GI home that are being patched, run the following commands as thehome owner to extract the OPatch utility.

 

$ unzip -d

$ /OPatch/opatch version

 

The version output of the previous command should be 12.1.0.1.7 or later.


--------------------------------------------------------------------------------------------------------------------------

Add the directory containing the opatchauto to the $PATH environment variable.

 

export PATH=$PATH:/u01/app/12.1.0/grid/OPatch

 

Backup the old and unzip the latest OPatch utility to ALL the HOMEs ( Grid Infrastructure and RAC ).

Replace OPatch on GRID_HOME

 

[root@IND120202 23273935]# cd /u01/app/12.1.0/grid

[root@IND120202 grid]# mv OPatch OPatch_bak

[root@IND120202 grid]# cd /mnt/hgfs/Shared_Folder/12.1.0.1/patches

[root@IND120201 patches]$ cp p6880880_121010_Linux-x86-64.zip /tmp

[root@IND120201 patches]$ cd /tmp

[root@IND120201 tmp]$ unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/12.1.0/grid
[root@IND120201 tmp]$ chown -R grid:oinstall /u01/app/12.1.0/grid/OPatch/

Replace OPatch on ORACLE_HOME

[root@IND120202 23273935]# cd /u01/app/oracle/product/12.1.0/db_1
[root@IND120202 db_1]# mv OPatch OPatch_bak
[root@IND120201 db_1]$ cd /tmp
[root@IND120201 tmp]$ unzip p6880880_121010_Linux-x86-64.zip -d /u01/app/oracle/product/12.1.0/db_1
[root@IND120201 tmp]$ chown -R oracle:oinstall /u01/app/oracle/product/12.1.0/db_1/OPatch/

 

Check the patch details

 

[grid@IND120201 OPatch]$ ./opatch lsinventory

 

Oracle Interim Patch Installer version 12.2.0.1.5

Copyright (c) 2016, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : /u01/app/12.1.0/grid

Central Inventory : /u01/app/oraInventory

from           : /u01/app/12.1.0/grid/oraInst.loc

OPatch version    : 12.2.0.1.5

OUI version       : 12.1.0.1.0

Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2016-08-07_21-51-20PM_1.log

 

Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-08-07_21-51-20PM.txt

 

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: IND120201.homedomain

ARU platform id: 226

ARU platform description:: Linux x86-64

 

Installed Top-level Products (1):

 

Oracle Grid Infrastructure 12c 12.1.0.1.0

There are 1 products installed in this Oracle Home.

 

 

There are no Interim patches installed in this Oracle Home.

 

 

Patch level status of Cluster nodes :

 

Patching Level Nodes

-------------- -----

405750136 ind120201

405750088 ind120202

 

--------------------------------------------------------------------------------

 

Be sure there are no conflicting patches in the above output. If so , proceed with below. If there are conflicting one-off patches, check the README document.

Installing the patch : Run this as root.


The node and the
Set the environment to the HOME ( GRID HOME here ) from which you are running the command.

[root@IND120201 ~]# ps -ef | grep pmon
grid 7005     1  0 22:16 ?        00:00:00 asm_pmon_+ASM1
oracle    8477 1  0 22:18 ?        00:00:00 ora_pmon_INTRAC03_1
root     20887 20850  0 22:39 pts/0    00:00:00 grep pmon
[root@IND120201 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/grid

 

[root@IND120201 OPatch]# ./opatchauto apply /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935

 

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2016-08-07_10-07-07PM.log.

 

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2016-08-07_10-07-42PM.log

The id for this session is JY5M

[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/db_1

[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0/grid

 

Executing OPatch prereq operations to verify patch applicability on CRS Home........

 

 

Executing OPatch prereq operations to verify patch applicability on RAC Home........

 

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0/grid successfully

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[init:init] Executing GIRACPrereqAction action on home /u01/app/oracle/product/12.1.0/db_1

[init:init] Executing GIRACPrereqAction action on home /u01/app/12.1.0/grid

 

Executing prereq operations before applying on CRS Home........

 

 

Executing prereq operations before applying on RAC Home........

 

[init:init] GIRACPrereqAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[init:init] GIRACPrereqAction action completed on home /u01/app/12.1.0/grid successfully

[init:init] Executing RACDBPrereqAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Executing prereq operations before applying on RAC Home........

 

[init:init] RACDBPrereqAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[shutdown:prepare-shutdown] Executing RACPrepareShutDownAction action on home /u01/app/oracle/product/12.1.0/db_1

 

No step execution required.........

 

[shutdown:prepare-shutdown] RACPrepareShutDownAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/12.1.0/grid

 

Performing prepatch operations on CRS Home........

 

Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_ind120201_2016-08-07_10-11-44PM.log

 

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/12.1.0/grid successfully

[offline:prepare] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Performing prepatch operation on RAC Home........

 

[offline:prepare] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Start applying binary patches on RAC Home........

 

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[offline:finalize] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Performing postpatch operation on RAC Home........

 

[offline:finalize] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/12.1.0/grid

 

Start applying binary patches on CRS Home........

 

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/12.1.0/grid successfully

[startup:startup] Executing GIStartupAction action on home /u01/app/12.1.0/grid

 

Performing postpatch operations on CRS Home........

 

Postpatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_ind120201_2016-08-07_10-26-04PM.log

 

[startup:startup] GIStartupAction action completed on home /u01/app/12.1.0/grid successfully

[startup:finalize] Executing RACFinalizeStartAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Preparing RAC Home after database service restarted........

 

[startup:finalize] RACFinalizeStartAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[online:product-patching] Executing RACOnlineAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Trying to apply SQL patches on RAC Home.

 

[WARNING] The database instance 'INTRAC03_1' from '/u01/app/oracle/product/12.1.0/db_1', in host'ind120201' is not running. SQL changes, if any,  will not be applied.

To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).

Refer to the readme to get the correct steps for applying the sql changes.

 

[online:product-patching] RACOnlineAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/12.1.0/grid

 

Verifying patches applied on CRS Home.

 

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/12.1.0/grid successfully

[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/oracle/product/12.1.0/db_1

 

Verifying patches applied on RAC Home.

 

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/oracle/product/12.1.0/db_1 successfully

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:ind120201

RAC Home:/u01/app/oracle/product/12.1.0/db_1

Summary:

 

==Following patches were SKIPPED:

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/17303297

Reason: This patch is not applicable to this specified target type - "rac_database"

 

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/17077442

Log: /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-07_22-13-24PM_1.log

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/23054354

Log: /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-07_22-13-24PM_1.log

 

 

Host:ind120201

CRS Home:/u01/app/12.1.0/grid

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/17077442

Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-07_22-17-55PM_1.log

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/17303297

Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-07_22-17-55PM_1.log

 

Patch: /mnt/hgfs/Shared_Folder/12.1.0.1/Patches/23273935/23054354

Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-07_22-17-55PM_1.log

 

SQL ERROR RESOLUTION

 

While installation we faced the below error.

 

WARNING] The database instance 'INTRAC03_1' from '/u01/app/oracle/product/12.1.0/db_1', in host'ind120201' is not running. SQL changes, if any,  will not be applied.

 

To resolve this, start the database and run below command to apply the SQL changes to the DB.

 

[oracle@IND120201 ~]$ ps -ef | grep pmon

grid 6911     1  0 17:13 ?        00:00:00 asm_pmon_+ASM1

oracle 15527     1  0 17:24 ?        00:00:00 ora_pmon_INTRAC03_1

oracle   29482 20832  0 17:46 pts/0    00:00:00 grep pmon

 

[oracle@IND120201 ~]$ . oraenv

ORACLE_SID = [oracle] ? INTRAC03

The Oracle base has been set to /u01/app/oracle

 

[oracle@IND120201 ~]$ export ORACLEA_SID=INTRAC03_1

 

[oracle@IND120201 ~]$ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1/ ORACLE_SID=INTRAC03_1 /u01/app/oracle/product/12.1.0/db_1/OPatch/datapatch

 

SQL Patching tool version 12.1.0.1.0 on Sun Aug 14 17:37:20 2016

Copyright (c) 2014, Oracle.  All rights reserved.

 

Connecting to database...OK

Determining current state...done

Adding patches to installation queue and performing prereq checks...done

Installation queue:

  Nothing to roll back

  The following patches will be applied: 19769486,20299016,20831107,21352619,21951844,22291141,23054354

Installing patches...

Patch installation complete.  Total patches installed: 7

Validating logfiles...done

SQL Patching tool complete on Sun Aug 14 17:46:22 2016

[oracle@IND120201 ~]$


Repeat this process on the 2nd node.

 

Happy Patching ...