Oracle 12.1.0.2.0 GI Management Repository Best Practices

Version 8
Visibility: Open to anyone

    Introduction:

    I faced some challenges during exploring new feature of 12c version 12.1.0.2.0 GIMR while deploying in Production.

    CHM Schema is not available in MGMTDB (12c GIMR) ?

    Re: Node Reboot Analysis using MGMTDB (12c GIMR) ?

    $ oclumon dumpnodeview -allnodes -last "12:00:00"

    CRS-9118-Grid Infrastructure Management Repository connection error

     

    $ oclumon dumpnodeview -allnodes -s "2016-10-13 14:01:29" -e "2016-10-13 14:50:41"

    CRS-9118-Grid Infrastructure Management Repository connection error

     

    mgmtca not working in 12c GIMR?

    $ mgmtca

    MGTCA-1101 : failure in Oracle Grid Infrastructure Management Repository operation

    *Cause: A SQL request to the Oracle Grid Infrastructure Management Repository encountered an error.

    *Action: Contact Oracle Support Services

    Environment:

    Oracle12c 2-Node RAC v12.1.0.2.0

    RHEL 6.8 64 bit

     

    root user:

    ==========

    <GI_HOME>/bin/crsctl stop res ora.crf -init

    <GI_HOME>/bin/crsctl modify res ora.crf -attr ENABLED=0 -init

    Grid User:

    ==========

    srvctl stop mgmtdb

    Create a CDB for the MGMTDB using DBCA

    <GI_HOME>/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB

    $/u01/app/12.1.0/grid/bin/dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +SASDATA1 -datafileJarLocation /u01/app/12.1.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck

     

     

    Create a PDB within the MGMTDB using DBCA.

     

    FIND YOUR CLUSTER NAME:

    -bash-4.1$ cemutlo -n

    blrpm-clstr

    NOTE: The CLUSTER_NAME needs to have any hyphens (“-“) replaced with underscores (“_”)

     

    $/u01/app/12.1.0/grid/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName blrpm_clstr -createPDBFrom RMANBACKUP -PDBBackUpfile /u01/app/12.1.0/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile /u01/app/12.1.0/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true

     

    Confirm the node on which MGMTDB is running by executing.

    srvctl status MGMTDB

     

    Execute below command from the Node on which MGMTDB is running

    mgmtca

     

    Root

    =====

    crsctl modify res ora.crf -attr ENABLED=1 -init

    crsctl start res ora.crf -init

     

    Grid User

    ============

    $ oclumon manage -get repsize

    CHM Repository Size = 136320 seconds

     

    $ oclumon manage -get reppath

    CHM Repository Path = +ORADATA/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdata.975.925137317

     

    #Three days data retention for 2 Node RAC

     

    -bash-4.1$ oclumon manage -repos checkretentiontime 259200

    The Cluster Health Monitor repository is too small for the desired retention. Please first resize the repository to 3896 MB

    -bash-4.1$ oclumon manage -repos changerepossize 3896

    The Cluster Health Monitor repository was successfully resized.The new retention is 259260 seconds.

     

    -bash-4.1$ oclumon manage -repos checkretentiontime 259200

    The Cluster Health Monitor repository can support the desired retention for 2 hosts

     

    -bash-4.1$ oclumon manage -get repsize

    CHM Repository Size = 259260 seconds

     

     

    #Fetch Data from Cluster Health Monitor - Repository

    oclumon dumpnodeview -allnodes -last "12:00:00"

     

    oclumon dumpnodeview -allnodes -s "2016-10-13 14:01:29" -e "2016-10-13 14:50:41"

     

     

     

     

    -bash-4.1$ oclumon dumpnodeview -allnodes -h

    *Where

    -allnodes        = Dump node views for all nodes

                     -s   = Specify start time for range dump of node views

                     -e   = Specify end time for range dump of node views           

    Absolute timestamp must be in "YYYY-MM-DD HH24:MI:SS"           

    format, for example "2007-11-12 23:05:00"

     

    -bash-4.1$ oclumon dumpnodeview -allnodes -s "2016-10-13 15:15:50" -e "2016-10-13 15:15:59"

    ----------------------------------------

    Node: pcrfnd1 Clock: '16-10-13 15.15.51 Asia/Calcutta' SerialNo:462

    ----------------------------------------

    SYSTEM:

    #pcpus: 2 #vcpus: 24 cpuht: Y chipname: Intel(R) cpu: 2.51 cpuq: 0 physmemfree: 2745556 physmemtotal: 66120612 mcache: 39928872 swapfree: 32767992 swaptotal: 32767992 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 704 iow: 312 ios: 136 swpin: 0 swpout: 0 pgin: 235 pgout: 164 netr: 92.261 netw: 38.991 procs: 1604 procsoncpu: 2 rtprocs: 83 rtprocsoncpu: N/A #fds: 59376 #sysfdlimit: 6815744 #disks: 17 #nics: 3 nicErrors: 0

     

    TOP CONSUMERS:

    topcpu: 'ora_rsm0_blrdb1(75191) 6.60' topprivmem: 'java(216225) 5750832' topshm: 'ora_dbw0_blrdb1(74904) 2089928' topfd: 'java(102729) 418' topthread: 'java(102729) 276'

     

    ----------------------------------------

    Node: pcrfnd1 Clock: '16-10-13 15.15.56 Asia/Calcutta' SerialNo:463

    ----------------------------------------

    SYSTEM:

    #pcpus: 2 #vcpus: 24 cpuht: Y chipname: Intel(R) cpu: 2.10 cpuq: 0 physmemfree: 2740368 physmemtotal: 66120612 mcache: 39928900 swapfree: 32767992 swaptotal: 32767992 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 326 iow: 463 ios: 78 swpin: 0 swpout: 0 pgin: 109 pgout: 210 netr: 52.314 netw: 75.884 procs: 1605 procsoncpu: 2 rtprocs: 83 rtprocsoncpu: N/A #fds: 59424 #sysfdlimit: 6815744 #disks: 17 #nics: 3 nicErrors: 0

     

    TOP CONSUMERS:

    topcpu: 'osysmond.bin(42779) 5.20' topprivmem: 'java(216225) 5750832' topshm: 'ora_dbw0_blrdb1(74904) 2089928' topfd: 'java(102729) 418' topthread: 'java(102729) 276'

     

    ----------------------------------------

    Node: pcrfnd2 Clock: '16-10-13 15.15.54 Asia/Calcutta' SerialNo:50650

    ----------------------------------------

    SYSTEM:

    #pcpus: 2 #vcpus: 24 cpuht: Y chipname: Intel(R) cpu: 1.18 cpuq: 0 physmemfree: 46625764 physmemtotal: 66120612 mcache: 10598984 swapfree: 32767992 swaptotal: 32767992 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 211 iow: 63 ios: 51 swpin: 0 swpout: 0 pgin: 71 pgout: 32 netr: 89.127 netw: 128.099 procs: 941 procsoncpu: 3 rtprocs: 83 rtprocsoncpu: N/A #fds: 26112 #sysfdlimit: 6815744 #disks: 16 #nics: 3 nicErrors: 0

     

    TOP CONSUMERS:

    topcpu: 'ora_rsm0_blrdb2(194693) 6.40' topprivmem: 'java(88622) 5548840' topshm: 'ora_mman_blrdb2(194443) 1315088' topfd: 'java(88622) 418' topthread: 'java(88622) 263'

     

    ----------------------------------------

    Node: pcrfnd2 Clock: '16-10-13 15.15.59 Asia/Calcutta' SerialNo:50651

    ----------------------------------------

    SYSTEM:

    #pcpus: 2 #vcpus: 24 cpuht: Y chipname: Intel(R) cpu: 1.10 cpuq: 0 physmemfree: 46626332 physmemtotal: 66120612 mcache: 10598996 swapfree: 32767992 swaptotal: 32767992 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 97 iow: 87 ios: 50 swpin: 0 swpout: 0 pgin: 33 pgout: 70 netr: 62.592 netw: 54.663 procs: 941 procsoncpu: 2 rtprocs: 83 rtprocsoncpu: N/A #fds: 26112 #sysfdlimit: 6815744 #disks: 16 #nics: 3 nicErrors: 0

     

    TOP CONSUMERS:

    topcpu: 'osysmond.bin(8902) 3.19' topprivmem: 'java(88622) 5548840' topshm: 'ora_mman_blrdb2(194443) 1315088' topfd: 'java(88622) 418' topthread: 'java(88622) 263'

    -bash-4.1$

    mgmtdb – Service relocation from Node-1 to Node-2

     

    RAC

    Configured RAC Node-2

    Directory

    [root@bsnlblrpmdb2 ~]# mkdir -p /backup/bcpscripts

    [root@bsnlblrpmdb2 ~]# chown -R orablr:oinstall /backup/bcpscripts

    [root@bsnlblrpmdb2 ~]# chmod -R 776 /backup/bcpscripts

     

    [root@bsnlblrpmdb2 ~]# mkdir -p /backup/bcpscriptslog

    [root@bsnlblrpmdb2 ~]# chown -R orablr:oinstall /backup/bcpscriptslog

    [root@bsnlblrpmdb2 ~]# chmod -R 776 /backup/bcpscriptslog

    Script Code

     

    [orablr@bsnlblrpmdb2 ~]$ vi /backup/bcpscripts/mgmtdb_relocation.sh

    export GRID_HOME=/u01/app/12.1.0/grid

    mgmtlog=`date '+/backup/blrdbscriptslogs/log_mgmtdb_relocation_%d%m%y.log'`

    date  >> $mgmtlog

    node1=$($GRID_HOME/bin/srvctl status mgmtdb | grep running | awk '{print $NF}')

    if [ $node1 = 'node2' ]; then

    echo "MGMTDB is already running on preffered node2"  >> $mgmtlog

    exit

     

    elif [ $node1 = 'node1' ]; then

    echo "MGMTDB is running on node1" >>  $mgmtlog

    echo "MGMTDB relocation starting.. node1 --> node2"  >> $mgmtlog

    $GRID_HOME/bin/srvctl relocate mgmtdb -node node2

    result=$($GRID_HOME/bin/srvctl status mgmtdb | grep running | awk '{print $NF}')

    if [ $result = 'bsnlblrpmdb2' ]; then

    echo "MGMTDB relocated successfully on node2"  >> $mgmtlog

    else

    echo "MGMTDB reloaction failed" >>  $mgmtlog

    fi

     

    else

    echo "Starting MGMTDB on node2..." >>  $mgmtlog

    $GRID_HOME/bin/srvctl start mgmtdb -node bsnlblrpmdb2

    result=$($GRID_HOME/bin/srvctl status mgmtdb | grep running | awk '{print $NF}')

    if [ $result = 'bsnlblrpmdb2' ]; then

    echo "MGMTDB started successfully on node2"  >> $mgmtlog

    else

    echo "MGMTDB startup failed" >>  $mgmtlog

    fi

    fi

    Permission

    chmod 776 mgmtdb_relocation.sh

    Cronjob

    [orablr@bsnlblrpmdb2 ~]$ crontab -l

    19 * * * * /backup/bcpscripts/mgmtdb_relocation.sh > /backup/bcpscripts/Cron_mgmtdb_relocation.log 2>&1

     

     

    References:

    How to Move/Recreate GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFSetc) (Doc ID 1589394.1)

    Managing the Cluster Health Monitor Repository (Doc ID 1921105.1)

    Node Reboot Analysis using MGMTDB (12c GIMR) ? https://community.oracle.com/message/14067199

    mgmtca not working in 12c GIMR? https://community.oracle.com/thread/3980784

     

    Author:

    Hitesh Gondalia

    Lead Database Administrator

    OCE RACv11g,OCP v12c/11g/10g/9i

    CloudDB OTN Community - Master Tag

    https://hiteshgondalia.wordpress.com