Apply RU JAN2020 to a 12.2 2-node RAC database (not GI) on AIX.

 

To patch my 2-node RAC database, I like to work with 4 putty sessions, 2 on each node:

 

The A ones are for actions and the B ones are for checks.  In other words, I will run all the commands that modify anything in the A puttys and run my checks in the B ones. That way I can easily get a history of the actions only, not mixed up with a thousand check commands.

I also like to have at hand the OEM page for the database I’m about to patch:

 

Run a precheck with OPatch:

opatch prereq CheckConflictAgainstOHWithDetail -ph /app/rac40/oracle/30593149

Oracle Interim Patch Installer version 12.2.0.1.19

Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /app/ora/oracle/product/12.2.0

Central Inventory : /app/oraInventory

from           : /app/ora/oracle/product/12.2.0/oraInst.loc

OPatch version    : 12.2.0.1.19

OUI version       : 12.2.0.1.4

Log file location : /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_09-52-26AM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.                 

                                                              

 

Check the patch can be applied with no downtime (rolling patch):

opatch query -is_rolling_patch /app/rac40/oracle/30593149

 

After having warned supervision and any other personnel, stop the instance of the 1st node:

node1@oracle:MD1:/home/oracle # srvctl stop instance -d MD -i MD1 -f ; srvctl status database -thishome

Database unique name: MD

Instance MD1 is not running on node node1

Instance MD2 is running on node node2

 

cd /app/rac40/oracle/30593149 ; $ORACLE_HOME/OPatch/opatch apply -report –verbose

“-report” means the patch won’t be applied but all the actions will be checked. You can run that with or without –verbose, which just makes the report more verbose. I use verbose when I run a patch for the 1st time then no longer use it subsequent times.

It will then ask for 2 human confirmations:

 

Running prerequisite check "CheckPatchApplyDependents" ...

All the required patch(es) are present in the Oracle Home.

Running prerequisite check "CheckMultipleTranslationPatches" ...

OPatch continues with these patches:   30593149

Do you want to proceed? [y|n]

y

User Responded with: Y

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/app/ora/oracle/product/12.2.0')

  Is the local system ready for patching? [y|n]

Y

 

Even though nothing was applied, it finished with

Done applying all patches.

Patch 30593149 successfully applied.

Log file location: /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_10-30-27AM_1.log

OPatch succeeded.

 

So then we can apply for real with no qualms:

$ORACLE_HOME/OPatch/opatch apply –verbose

 

Again it asks for 2 confirmations:

OPatch continues with these patches:   30593149

Do you want to proceed? [y|n]

Y

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/app/ora/oracle/product/12.2.0')

  Is the local system ready for patching? [y|n]

Y

 

In a B putty, I check that instance 1 is stopped with:

# srvctl status database -thisversion

Database unique name: MD

Instance MD1 is not running on node node1

Instance MD2 is running on node node2

The application of the patch finishes successfully with warnings:

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xargv

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xrcfg

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xrc

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

Make Action done

Patch 30593149 successfully applied.

OPatch Session completed with warnings.

Log file location: /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_10-37-54AM_1.log

OPatch completed with warnings.

 

The warnings are those above. I’ve had them on several systems and so far so good.

Now let’s go to the second node.

 

Let’s check that OPatch has the required minimum version (see the patch readme.html to know which minimum version of OPatch you MUST have)

So let’s use the “node2 B” putty:

node2@oracle:MD2:/home/oracle # cd $ORACLE_HOME; ./OPatch/opatch version

OPatch Version: 12.2.0.1.19

OPatch succeeded.

 

node2@oracle:MD2:/home/oracle # srvctl status database -d MD

Instance MD1 is not running on node node1

Instance MD2 is running on node node2

 

Let’s start instance one from the “node 2 A” putty:

srvctl start instance -db MD -i MD1

 

from the “node 2 B” putty, let’s check the situation:

node2@oracle:MD2:/tmp # srvctl status database -thishome

Database unique name: MD

Instance MD1 is running on node node1

Instance MD2 is running on node node2

 

From “node2 A”, let’s stop instance2:

srvctl stop instance -db MD -i MD2

 

 

from the “node 2 B” putty, let’s check the situation:

 

node2@oracle:MD2:/tmp # srvctl status database -thisversion

Database unique name: MD

Instance MD1 is running on node node1

Instance MD2 is not running on node node2

 

From “node2 A”, let’s apply the patch (first you should run a dry-run with –report if this is your 1st cluster ). Here I don’t use –verbose as the patch ran OK on node 1, and I use –silent so as to avoid manually entering “y” as we did on the 1stnode :

 

cd /app/rac40/oracle/30593149 ; $ORACLE_HOME/OPatch/opatch apply –silent

 

With no human intervention to confirm anything, the patch is successfully applied to node2 with the same warnings as on node1:

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xargv

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xrcfg

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

ld: 0711-773 WARNING: Object /lib/crt0_64.o, imported symbol p_xrc

        Symbol was expected to be local. Extra instructions

        are being generated to reference the symbol.

 

Patch 30593149 successfully applied.

OPatch Session completed with warnings.

Log file location: /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_11-11-23AM_1.log

OPatch completed with warnings.

 

In “node2 B”, I count the number of occurrences of the word “error” (lower or upper case) in the log:

grep -ci ErroR /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_10-17-16AM_1.log

and I look at those occurences to make sure they’re expected (the “error” character string appears in many messages but does not mean there was an error in the application of the RU):

view +/error /app/ora/oracle/product/12.2.0/cfgtoollogs/opatch/opatch2020-04-15_10-17-16AM_1.log

in vi, I “:set ic” to make my search case-insentsitive then I search for “ErroR” for example, and it shows me every occurrence of that character string regardless of case.

Then I restart instance2:

srvctl start instance -db MD -i MD2 ; srvctl status –thisversion

 

And I check that opatch lspatches does mention patch 30593149

node2@oracle:MD2:/tmp # $ORACLE_HOME/OPatch/opatch lspatches

30593149;Database Jan 2020 Release Update : 12.2.0.1.200114 (30593149)

OPatch succeeded.

 

So I go back to node1 to patch the database:

 

node1@oracle:MD1:/app/rac40/oracle/30593149 # cd $ORACLE_HOME/OPatch ;  ./datapatch -verbose

SQL Patching tool version 12.2.0.1.0 Production on Wed Apr 15 11:58:41 2020

Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /app/ora/oracle/cfgtoollogs/sqlpatch/sqlpatch_46727586_2020_04_15_11_58_41/sqlpatch_invocation.log

Connecting to database...OK

Bootstrapping registry and package to current versions...done

Determining current state...done

Current state of SQL patches:

Bundle series DBRU:

ID 200114 in the binary registry and not installed in the SQL registry

Adding patches to installation queue and performing prereq checks...

Installation queue:

Nothing to roll back

The following patches will be applied:

30593149 (DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114)

Installing patches...  

 

After about 6mn in my case, I get this:

Patch installation complete.  Total patches installed: 1

 

Validating logfiles...

Patch 30593149 apply: SUCCESS

logfile: /app/ora/oracle/cfgtoollogs/sqlpatch/30593149/23339232/30593149_apply_MD_2020Apr15_12_00_19.log (no errors)

SQL Patching tool complete on Wed Apr 15 12:04:34 2020

 

Then I carry out a few checks:

  15-04 12:13 MD2 SYS AS SYSDBA> SELECT patch_id,DESCRIPTION,status,action,action_time  FROM dba_registry_sqlpatch order by action_time desc;

PATCH_ID DESCRIPTION STATUS     ACTION    ACTION_TIME

---------- ----------------------------------------------------------- ---------- --------- ---------------------------------------------------------------------------

30593149 DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114            SUCCESS    APPLY 15/04/20 12:04:33,819233

30593149 DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114            SUCCESS    ROLLBACK 02/04/20 12:57:51,864195

30593149 DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114            SUCCESS    APPLY 20/03/20 16:36:59,186335

25171037 DATABASE PATCH SET UPDATE 12.1.0.2.170418                   SUCCESS    APPLY 27/08/17 11:33:42,719191

21359755 Database Patch Set Update : 12.1.0.2.5 (21359755)           SUCCESS    APPLY 25/01/16 08:52:42,951516

21555660 Database PSU 12.1.0.2.5, Oracle JavaVM Component (Oct2015)  SUCCESS APPLY     25/01/16 08:52:42,641226

 

During the whole process, I also have a window with OEM open on that database to check that at least 1 instance is running:

Please do not hesitate to comment on my method in the comments below, or tell me what could be improved, or share how you do it -- we can all become better together.