1 2 Previous Next 21 Replies Latest reply on Sep 19, 2016 8:35 PM by Junaidullah.Khan-Oracle

    RAC database will not open after enabling DB Vault

    Junaidullah.Khan-Oracle

      11.2.0.4

      Linux x86-64

      SLES-11


      After enabling and configuring Database Vault, we can not open our RAC database.

      The alert.log shows the following:

      Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee1/trace/lnkdbee1_ora_13568.trc:
      ORA-00604: error occurred at recursive SQL level 1
      ORA-00942: table or view does not exist
      Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee1/trace/lnkdbee1_ora_13568.trc:
      ORA-00604: error occurred at recursive SQL level 1
      ORA-00942: table or view does not exist
      Error 604 happened during db open, shutting down database
      USER (ospid: 13568): terminating the instance due to error 604
      Instance terminated by USER, pid = 13568
      ORA-1092 signalled during: ALTER DATABASE OPEN...
      opiodr aborting process unknown ospid (13568) as a result of ORA-1092
      Wed Jun 22 14:32:48 2016
      ORA-1092 : opitsk aborting process

      And then, abort the startup process.


       

      If Database Vault disabled, the Database starts normally.

       

      Things that have been tried thus far:


      1) Enabling DB Vault is in line with following doc:

      http://docs.oracle.com/cd/B28359_01/server.111/b31222/dvdisabl.htm#BAJFJFFGhttp://docs.oracle.com/cd/B28359_01/server.111/b31222/dvdisabl.htm#BAJFJFFG

       

      2) reinstall the XDB component using the procedure from the following Metalink note:

      https://mosemp.us.oracle.com/epmos/faces/ui/km/DocumentDisplay.jspx?id=1292089.1Note 1292089.1 - XDB 11g Install and Deinstall Note


      After reinstall XDB 11g (following the procedure) we have the component INVALID:

      SQL> select comp_name, version, status
      from dba_registry
      where comp_id = 'XDB';  2    3

      COMP_NAME                            VERSION      STATUS
      ------------------------------------ ------------ --------
      Oracle XML Database                  11.2.0.4.0   INVALID

      SQL> select owner, object_name, object_type, status
      from dba_objects
      where status = 'INVALID'
      and owner in ('SYS', 'XDB');  2    3    4

      OWNER        OBJECT_NAME                         OBJECT_TYPE         STATUS
      ------------ ----------------------------------- ------------------- --------
      SYS          DBMS_NETWORK_ACL_ADMIN              PACKAGE BODY        INVALID
      SYS          DBMS_XS_PRINCIPAL_EVENTS_INT        PACKAGE BODY        INVALID
      SYS          XS$CATVIEW_UTIL                     PACKAGE BODY        INVALID
      SYS          AWM_CREATEXDSFOLDER                 FUNCTION            INVALID
      SYS          DBMS_CUBE_EXP                       PACKAGE BODY        INVALID
      SYS          DBMS_METADATA                       PACKAGE BODY        INVALID
      XDB          DBMS_RESCONFIG                      PACKAGE BODY        INVALID
      XDB          DBMS_XMLSCHEMA                      PACKAGE BODY        INVALID
      XDB          DBMS_CSX_INT                        PACKAGE BODY        INVALID
      XDB          DBMS_XDBUTIL_INT                    PACKAGE BODY        INVALID
      XDB          DBMS_XDB                            PACKAGE BODY        INVALID
      XDB          DBMS_XDBRESOURCE                    PACKAGE BODY        INVALID
      XDB          DBMS_XSLPROCESSOR                   PACKAGE BODY        INVALID
      XDB          DBMS_XMLPARSER                      PACKAGE BODY        INVALID
      XDB          DBMS_XMLDOM                         PACKAGE BODY        INVALID

      XDB          XIMETADATA_PKG                      PACKAGE BODY        INVALID
      XDB          DBMS_XDBT                           PACKAGE BODY        INVALID

      17 rows selected.

      SQL>


      The list of the invalid components is the following:

      COMP_ID,COMP_NAME,VERSION,STATUS
      ORDIM,Oracle Multimedia,11.2.0.4.0,INVALID
      XOQ,Oracle OLAP API,11.2.0.4.0,INVALID
      SDO,Spatial,11.2.0.4.0,INVALID
      APEX,Oracle Application Express,3.2.1.00.12,INVALID
      XDB,Oracle XML Database,11.2.0.4.0,INVALID


      3) After removing all invalid components, we're enabled DV again, and we hit the same issue:

      [RACDESAWEB-nodo2]oracle@LNKI942:~/util > srvctl start db -d lnkdbee
      PRCR-1079 : Failed to start resource ora.lnkdbee.db
      CRS-5017: The resource action "ora.lnkdbee.db start" encountered the following error:
      ORA-01092: ORACLE instance terminated. Disconnection forced
      ORA-00604: error occurred at recursive SQL level 1
      ORA-00942: table or view does not exist
      Process ID: 2169
      Session ID: 16 Serial number: 1
      . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4/grid/log/lnki942/agent/crsd/oraagent_oracle//oraagent_oracle.log".

      CRS-5017: The resource action "ora.lnkdbee.db start" encountered the following error:
      ORA-01092: ORACLE instance terminated. Disconnection forced
      ORA-00604: error occurred at recursive SQL level 1
      ORA-00942: table or view does not exist
      Process ID: 40200
      Session ID: 16 Serial number: 1
      . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4/grid/log/lnki941/agent/crsd/oraagent_oracle//oraagent_oracle.log".

      CRS-2674: Start of 'ora.lnkdbee.db' on 'lnki942' failed
      CRS-2674: Start of 'ora.lnkdbee.db' on 'lnki941' failed
      CRS-2632: There are no more servers to try to place resource 'ora.lnkdbee.db' on that would satisfy its placement policy
      [RACDESAWEB-nodo2]oracle@LNKI942:~/util >

      ALERT.LOG:

      Wed Jul 13 13:30:45 2016
      SMON: enabling cache recovery
      ARC1: Becoming the 'no FAL' ARCH
      ARC1: Becoming the 'no SRL' ARCH
      ARC2: Becoming the heartbeat ARCH
      ARC3: Archival started
      ARC0: STARTING ARCH PROCESSES COMPLETE
      Archived Log entry 9885 added for thread 2 sequence 4880 ID 0xcda746ed dest 1:
      [2169] Successfully onlined Undo Tablespace 5.
      Undo initialization finished serial:0 start:3686137376 end:3686137756 diff:380 (3 seconds)
      Verifying file header compatibility for 11g tablespace encryption..
      Verifying 11g file header compatibility for tablespace encryption completed
      SMON: enabling tx recovery
      Database Characterset is WE8MSWIN1252
      No Resource Manager plan active
      Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_2169.trc:
      ORA-00604: error occurred at recursive SQL level 1

      ORA-00942: table or view does not exist

      Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_2169.trc:

      ORA-00604: error occurred at recursive SQL level 1

      ORA-00942: table or view does not exist

      Error 604 happened during db open, shutting down database

      USER (ospid: 2169): terminating the instance due to error 604

      Instance terminated by USER, pid = 2169

      ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {2:52275:204} */...

      opiodr aborting process unknown ospid (2169) as a result of ORA-1092

      Wed Jul 13 13:30:46 2016

      ORA-1092 : opitsk aborting process

       

      Regards,

      Junaid

        • 1. Re: RAC database will not open after enabling DB Vault
          toonie

          Hi Junaid,

           

          I don't know what your particular issue might be but you could enable tracing on the 'ORA-00942: table or view does not exist' error to get some more information as to what the complaint is about? This should generate a trace file in the dump directory.

           

          SQL> alter system set event '942 trace name errorstack forever ,level 3 ;'


          Why did you go down the route of reinstalling XDB?  Was it a particular error or? I just don't understand where that fits into the scenario above and why you though that would resolve the issue. Was it simply because you noted those invalid objects?

           

          Also - what does the trace file say?: /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_2169.trc


          Regards

          Ruan

          1 person found this helpful
          • 2. Re: RAC database will not open after enabling DB Vault
            Junaidullah.Khan-Oracle

            Initially I thought running the utlrp.sql script would do the trick, not realizing that that wouldn't help with invalid components- based on the errors that were noted from that script I went down the path of trying to reinstall XDB...in hindsight, i was off the mark there.

             

            Will update with the trace info shortly...

            • 3. Re: RAC database will not open after enabling DB Vault
              Junaidullah.Khan-Oracle

              two separate traces for your review, any insight or guidance would be much appreciated

               

              Trace file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee1/trace/lnkdbee1_ora_64038.trc

              Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

              With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,

              OLAP, Data Mining and Oracle Database Vault options

              ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1

              System name: Linux

              Node name: LNKI941

              Release: 3.0.101-65-default

              Version: #1 SMP Mon Sep 7 19:21:52 UTC 2015 (bef9feb)

              Machine: x86_64

              VM name: VMWare Version: 6

              Instance name: lnkdbee1

              Redo thread mounted by this instance: 1

              Oracle process number: 32

              Unix process pid: 64038, image: oracle@LNKI941 (TNS V1-V3)

               

               

              *** 2016-07-18 12:16:31.989

              *** SESSION ID:(16.5) 2016-07-18 12:16:31.989

              *** CLIENT ID:() 2016-07-18 12:16:31.989

              *** SERVICE NAME:() 2016-07-18 12:16:31.989

              *** MODULE NAME:(sqlplus@LNKI941 (TNS V1-V3)) 2016-07-18 12:16:31.989

              *** ACTION NAME:() 2016-07-18 12:16:31.989

               

              2016-07-18 12:16:31.988696 : Start recovery for domain=0, valid=0, flags=0x4

              Successfully allocated 2 recovery slaves

              Using 67 overflow buffers per recovery slave

              Thread 2 checkpoint: logseq 4919, block 9283, scn 302561478

                  on-disk rba: logseq 4919, block 9284, scn 302561480

                start recovery at logseq 4919, block 9284, scn 302561480

               

              *** 2016-07-18 12:16:32.256

              Started resilvering redo thread 2 seq 4919 blocks 9284-9333

               

              *** 2016-07-18 12:16:32.256

              Completed resilvering redo thread 2 seq 4919

               

              *** 2016-07-18 12:16:32.256

              Started writing zeroblks thread 2 seq 4919 blocks 9334-9341

               

              *** 2016-07-18 12:16:32.257

              Completed writing zeroblks thread 2 seq 4919

              ==== Redo read statistics for thread 2 ====

              Total physical reads (from disk and memory): 4096Kb

              -- Redo read_disk statistics --

              Read rate (ASYNC): 25Kb in 0.10s => 0.24 Mb/sec

              Longest record: 0Kb, moves: 0/70 (0%)

              Longest LWN: 1Kb, moves: 0/30 (0%), moved: 0Mb

              Last redo scn: 0x0000.1208b904 (302561540)

              ----------------------------------------------

              ----- Recovery Hash Table Statistics ---------

              Hash table buckets = 262144

              Longest hash chain = 1

              Average hash chain = 24/24 = 1.0

              Max compares per lookup = 1

              Avg compares per lookup = 96/120 = 0.8

              ----------------------------------------------

               

              *** 2016-07-18 12:16:32.259

              KCRA: start recovery claims for 24 data blocks

               

              *** 2016-07-18 12:16:32.276

              KCRA: blocks processed = 24/24, claimed = 24, eliminated = 0

               

              *** 2016-07-18 12:16:32.302

              Recovery of Online Redo Log: Thread 2 Group 5 Seq 4919 Reading mem 0

               

              *** 2016-07-18 12:16:32.305

              Completed redo application of 0.02MB

               

              *** 2016-07-18 12:16:32.307

              Completed recovery checkpoint

              ----- Recovery Hash Table Statistics ---------

              Hash table buckets = 262144

              Longest hash chain = 1

              Average hash chain = 24/24 = 1.0

              Max compares per lookup = 1

              Avg compares per lookup = 120/120 = 1.0

              ----------------------------------------------

              Recovery sets nab of thread 2 seq 4919 to 9334 with 8 zeroblks

              2016-07-18 12:16:32.584324 : Validate domain 0

              2016-07-18 12:16:32.584767 : Validated domain 0, flags = 0x0

               

              *** 2016-07-18 12:16:32.704

              Pick closed redo thread 2 to enable my thread

              Initial buffer sizes: read 1024K, overflow 832K, change 805K

               

              *** 2016-07-18 12:16:34.869

              ORA-00604: error occurred at recursive SQL level 1

              ORA-00942: table or view does not exist

              ORA-00604: error occurred at recursive SQL level 1

              ORA-00942: table or view does not exist

               

              *** 2016-07-18 12:16:34.869

              USER (ospid: 64038): terminating the instance due to error 604

               

              --------------------------------------------------------------------------------------------------------------------------

               

              Trace file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_39201.trc

              Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

              With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,

              OLAP, Data Mining and Oracle Database Vault options

              ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1

              System name: Linux

              Node name: LNKI942

              Release: 3.0.101-65-default

              Version: #1 SMP Mon Sep 7 19:21:52 UTC 2015 (bef9feb)

              Machine: x86_64

              VM name: VMWare Version: 6

              Instance name: lnkdbee2

              Redo thread mounted by this instance: 2

              Oracle process number: 32

              Unix process pid: 39201, image: oracle@LNKI942 (TNS V1-V3)

               

               

              *** 2016-07-18 12:09:21.338

              *** SESSION ID:(16.5) 2016-07-18 12:09:21.338

              *** CLIENT ID:() 2016-07-18 12:09:21.338

              *** SERVICE NAME:(SYS$USERS) 2016-07-18 12:09:21.338

              *** MODULE NAME:(sqlplus@LNKI942 (TNS V1-V3)) 2016-07-18 12:09:21.338

              *** ACTION NAME:() 2016-07-18 12:09:21.338

               

              ORA-00604: error occurred at recursive SQL level 1

              ORA-00942: table or view does not exist

              ORA-00604: error occurred at recursive SQL level 1

              ORA-00942: table or view does not exist

               

              *** 2016-07-18 12:09:21.339

              USER (ospid: 39201): terminating the instance due to error 604

              • 4. Re: RAC database will not open after enabling DB Vault
                toonie

                I'm afraid there is very little info in those trace files that I can decipher.

                 

                You didn't enable tracing on the 942 event prior to getting those trace files though? The 942 event tracing should mention the SQL involved, hence the 'missing' table.

                 

                Regards

                • 5. Re: RAC database will not open after enabling DB Vault
                  Junaidullah.Khan-Oracle

                  Ok, will try again.  What level should I set?

                   

                  alter system set events="942 trace name ERRORSTACK level 3"

                   

                  or

                   

                  alter system set events="942 dump errorstack level 1";

                  • 6. Re: RAC database will not open after enabling DB Vault
                    toonie

                    Try this:

                     

                    SQL> alter system set events '942  trace name ERRORSTACK level 3';

                    or

                    SQL> ALTER SYSTEM SET events = '942 TRACE NAME ERRORSTACK LEVEL 3';

                    • 7. Re: RAC database will not open after enabling DB Vault
                      Junaidullah.Khan-Oracle

                      I was pointed to a older trace file that exists but its HUGE- wouldn't want to post something too long here unless you say so....

                      Can I perhaps provide a limited portion of it perhaps? please specify.

                       

                      I did come across this:

                      ORA-00604: error occurred at recursive SQL level 1

                      ORA-04063: package body "XDB.DBMS_CSX_INT" has errors <-----could this be an area that requires attention ?

                      • 9. Re: RAC database will not open after enabling DB Vault
                        Junaidullah.Khan-Oracle

                        Same errors trying to start up the database

                        Mon Aug 01 13:18:12 2016
                        minact-scn: Inst 2 is now the master inc#:5 mmon proc-id:8888 status:0x7
                        minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
                        minact-scn: master found reconf/inst-rec before recscn scan old-inc#:5 new-inc#:5
                        parallel recovery started with 2 processes
                        Started redo scan
                        Completed redo scan
                        read 25 KB redo, 24 data blocks need recovery
                        Started redo application at
                        Thread 1: logseq 5163, block 29247, scn 308565661
                        Recovery of Online Redo Log: Thread 1 Group 3 Seq 5163 Reading mem 0
                          Mem# 0: +DATA/lnkdbee/onlinelog/group_3.442.859897527
                          Mem# 1: +DATA/lnkdbee/onlinelog/group_3.452.859897527
                        Completed redo application of 0.02MB
                        Completed instance recovery at
                        Thread 1: logseq 5163, block 29298, scn 308585850
                        21 data blocks read, 24 data blocks written, 25 redo k-bytes read
                        Thread 1 advanced to log sequence 5164 (thread recovery)
                        Redo thread 1 internally disabled at seq 5164 (SMON)
                        Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_8948.trc:
                        ORA-00604: error occurred at recursive SQL level 1
                        ORA-00942: table or view does not exist
                        Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_8948.trc:
                        ORA-00604: error occurred at recursive SQL level 1
                        ORA-00942: table or view does not exist
                        Error 604 happened during db open, shutting down database
                        USER (ospid: 8948): terminating the instance due to error 604
                        Instance terminated by USER, pid = 8948
                        ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:26993:764} */...
                        opiodr aborting process unknown ospid (8948) as a result of ORA-1092
                        Mon Aug 01 13:18:14 2016
                        ORA-1092 : opitsk aborting process

                        • 10. Re: RAC database will not open after enabling DB Vault
                          toonie

                          Can you post the content of this trace file?

                          Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee2/trace/lnkdbee2_ora_8948.trc:

                          I still think you need to enable system event trace on the 942 message and find out what table/view is being complained about. I believe this is the quickest way you will get to the solution of your issue.

                           

                          Regards

                          • 12. Re: RAC database will not open after enabling DB Vault
                            Junaidullah.Khan-Oracle

                            Mon Jul 18 11:49:24 2016

                            OS Pid: 51988 executed alter system set events '942 TRACE NAME ERRORSTACK LEVEL 3'

                            Mon Jul 18 11:52:42 2016

                            OS Pid: 51988 executed alter system set events '942 TRACE NAME ERRORSTACK FOREVER, LEVEL 3'

                            Mon Jul 18 11:54:48 2016

                            Shutting down instance (immediate)

                            Stopping background process SMCO

                            Shutting down instance: further logons disabled

                            Stopping background process QMNC

                            Stopping background process MMNL

                            Mon Jul 18 11:54:50 2016

                            minact-scn: Inst 1 is now the master inc#:4 mmon proc-id:51127 status:0x7

                            minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.1208b683 gcalc-scn:0x0000.1208b691

                            minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:4 errcnt:0

                            Stopping background process MMON

                            License high water mark = 33

                            All dispatchers and shared servers shutdown

                            ALTER DATABASE CLOSE NORMAL /* db agent *//* {1:64358:13229} */

                            Mon Jul 18 11:54:55 2016

                            SMON: disabling tx recovery

                            Stopping background process RCBG

                            SMON: disabling cache recovery

                            Mon Jul 18 11:54:59 2016

                            NOTE: Deferred communication with ASM instance

                            NOTE: deferred map free for map id 55

                            Redo thread 1 internally disabled at seq 5045 (LGWR)

                            Shutting down archive processes

                            Archiving is disabled

                            Mon Jul 18 11:54:59 2016

                            ARCH shutting down

                            Mon Jul 18 11:54:59 2016

                            ARCH shutting down

                            ARC2: Archival stopped

                            ARC0: Archival stopped

                            Mon Jul 18 11:54:59 2016

                            ARCH shutting down

                            ARC1: Archival stopped

                            Mon Jul 18 11:54:59 2016

                            ARC3: Archiving disabled thread 1 sequence 5045

                            Archived Log entry 9963 added for thread 1 sequence 5045 ID 0xcda746ed dest 1:

                            ARCH shutting down

                            ARC3: Archival stopped

                            Thread 1 closed at log sequence 5045

                            Successful close of redo thread 1

                            Mon Jul 18 11:55:00 2016

                            NOTE: Deferred communication with ASM instance

                            NOTE: deferred map free for map id 4

                            Mon Jul 18 11:55:00 2016

                            Completed: ALTER DATABASE CLOSE NORMAL /* db agent *//* {1:64358:13229} */

                            ALTER DATABASE DISMOUNT /* db agent *//* {1:64358:13229} */

                            Shutting down archive processes

                            Archiving is disabled

                            Mon Jul 18 11:55:00 2016

                            NOTE: Deferred communication with ASM instance

                            NOTE: deferred map free for map id 2

                            Completed: ALTER DATABASE DISMOUNT /* db agent *//* {1:64358:13229} */

                            ARCH: Archival disabled due to shutdown: 1089

                            Shutting down archive processes

                            Archiving is disabled

                            Mon Jul 18 11:55:01 2016

                            NOTE: force a map free for map id 2

                            NOTE: force a map free for map id 4

                            NOTE: force a map free for map id 55

                            ARCH: Archival disabled due to shutdown: 1089

                            Shutting down archive processes

                            Archiving is disabled

                            NOTE: force a map free for map id 498

                            NOTE: force a map free for map id 497

                            Mon Jul 18 11:55:02 2016

                            NOTE: Shutting down MARK background process

                            Mon Jul 18 11:55:02 2016

                            Stopping background process VKTM

                            Mon Jul 18 11:55:03 2016

                            freeing rdom 0

                            Mon Jul 18 11:55:09 2016

                            Instance shutdown complete

                            Mon Jul 18 12:16:22 2016

                            Starting ORACLE instance (normal)

                            ************************ Large Pages Information *******************

                            Per process system memlock (soft) limit = 64 KB

                             

                            Total Shared Global Region in Large Pages = 0 KB (0%)

                             

                            Large Pages used by this instance: 0 (0 KB)

                            Large Pages unused system wide = 0 (0 KB)

                            Large Pages configured system wide = 0 (0 KB)

                            Large Page size = 2048 KB

                             

                            RECOMMENDATION:

                              Total System Global Area size is 2370 MB. For optimal performance,

                              prior to the next instance restart:

                              1. Increase the number of unused large pages by

                            at least 1185 (page size 2048 KB, total size 2370 MB) system wide to

                              get 100% of the System Global Area allocated with large pages

                              2. Large pages are automatically locked into physical memory.

                            Increase the per process memlock (soft) limit to at least 2378 MB to lock

                            100% System Global Area's large pages into physical memory

                            ********************************************************************

                            LICENSE_MAX_SESSION = 0

                            LICENSE_SESSIONS_WARNING = 0

                            Initial number of CPU is 2

                            Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.

                              [name='eth1:1', type=1, ip=169.254.60.85, mac=00-50-56-99-7e-84, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]

                            Public Interface 'eth0' configured from GPnP for use as a public interface.

                              [name='eth0', type=1, ip=172.16.0.216, mac=00-50-56-99-7a-a3, net=172.16.0.0/23, mask=255.255.254.0, use=public/1]

                            Public Interface 'eth0:1' configured from GPnP for use as a public interface.

                              [name='eth0:1', type=1, ip=172.16.0.217, mac=00-50-56-99-7a-a3, net=172.16.0.0/23, mask=255.255.254.0, use=public/1]

                            Public Interface 'eth0:4' configured from GPnP for use as a public interface.

                              [name='eth0:4', type=1, ip=172.16.0.221, mac=00-50-56-99-7a-a3, net=172.16.0.0/23, mask=255.255.254.0, use=public/1]

                            Picked latch-free SCN scheme 3

                            Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST

                            WARNING: db_recovery_file_dest is same as db_create_file_dest

                            Autotune of undo retention is turned on.

                            LICENSE_MAX_USERS = 0

                            SYS auditing is disabled

                            Starting up:

                            Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

                            With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,

                            Data Mining and Oracle Database Vault options.

                            ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1

                            System name: Linux

                            Node name: LNKI941

                            Release: 3.0.101-65-default

                            Version: #1 SMP Mon Sep 7 19:21:52 UTC 2015 (bef9feb)

                            Machine: x86_64

                            VM name: VMWare Version: 6

                            Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/initlnkdbee1.ora

                            System parameters with non-default values:

                              processes                = 300

                              sessions                 = 500

                              spfile                   = "+DATA/lnkdbee/spfilelnkdbee.ora"

                              sga_target               = 2368M

                              control_files            = "+DATA/lnkdbee/controlfile/current.494.859893425"

                              control_files            = "+DATA/lnkdbee/controlfile/current.574.859893425"

                              db_block_size            = 8192

                              compatible               = "11.2.0.4.0"

                              cluster_database         = TRUE

                              db_create_file_dest      = "+DATA"

                              db_recovery_file_dest    = "+DATA"

                              db_recovery_file_dest_size= 5727M

                              thread                   = 1

                              undo_tablespace          = "UNDOTBS1"

                              instance_number          = 1

                              recyclebin               = "OFF"

                              db_securefile            = "PERMITTED"

                              os_roles                 = FALSE

                              remote_login_passwordfile= "EXCLUSIVE"

                              audit_sys_operations     = FALSE

                              db_domain                = "redlink.com.ar"

                              dispatchers              = "(PROTOCOL=TCP) (SERVICE=lnkdbeeXDB)"

                              remote_listener          = "clusterdesaweb-scan:1521"

                              job_queue_processes      = 0

                              audit_file_dest          = "/u01/app/oracle/admin/lnkdbee/adump"

                              audit_trail              = "DB"

                              db_name                  = "LNKDBEE"

                              open_cursors             = 300

                              sql92_security           = FALSE

                              pga_aggregate_target     = 787M

                              diagnostic_dest          = "/u01/app/oracle"

                            Cluster communication is configured to use the following interface(s) for this instance

                              169.254.60.85

                            cluster interconnect IPC version:Oracle UDP/IP (generic)

                            IPC Vendor 1 proto 2

                            Mon Jul 18 12:16:22 2016

                            PMON started with pid=2, OS id=63852

                            Mon Jul 18 12:16:22 2016

                            PSP0 started with pid=3, OS id=63856

                            Mon Jul 18 12:16:23 2016

                            VKTM started with pid=4, OS id=63864

                            VKTM running at (100ms) precision

                            Mon Jul 18 12:16:23 2016

                            GEN0 started with pid=5, OS id=63870

                            Mon Jul 18 12:16:23 2016

                            DIAG started with pid=6, OS id=63874

                            Mon Jul 18 12:16:24 2016

                            DBRM started with pid=7, OS id=63878

                            Mon Jul 18 12:16:24 2016

                            PING started with pid=8, OS id=63882

                            Mon Jul 18 12:16:24 2016

                            ACMS started with pid=9, OS id=63886

                            Mon Jul 18 12:16:24 2016

                            DIA0 started with pid=10, OS id=63890

                            Mon Jul 18 12:16:24 2016

                            LMON started with pid=11, OS id=63894

                            Mon Jul 18 12:16:24 2016

                            LMD0 started with pid=12, OS id=63898

                            * Load Monitor used for high load check

                            * New Low - High Load Threshold Range = [1920 - 2560]

                            Mon Jul 18 12:16:24 2016

                            LMS0 started with pid=13, OS id=63902

                            Mon Jul 18 12:16:24 2016

                            RMS0 started with pid=14, OS id=63908

                            Mon Jul 18 12:16:24 2016

                            LMHB started with pid=15, OS id=63912

                            Mon Jul 18 12:16:24 2016

                            MMAN started with pid=16, OS id=63916

                            Mon Jul 18 12:16:24 2016

                            DBW0 started with pid=17, OS id=63920

                            Mon Jul 18 12:16:24 2016

                            LGWR started with pid=18, OS id=63924

                            Mon Jul 18 12:16:24 2016

                            CKPT started with pid=19, OS id=63928

                            Mon Jul 18 12:16:24 2016

                            SMON started with pid=20, OS id=63932

                            Mon Jul 18 12:16:24 2016

                            RECO started with pid=21, OS id=63936

                            Mon Jul 18 12:16:24 2016

                            RBAL started with pid=22, OS id=63940

                            Mon Jul 18 12:16:24 2016

                            ASMB started with pid=23, OS id=63944

                            Mon Jul 18 12:16:24 2016

                            MMON started with pid=24, OS id=63948

                            Mon Jul 18 12:16:24 2016

                            MMNL started with pid=25, OS id=63954

                            starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

                            starting up 1 shared server(s) ...

                            NOTE: initiating MARK startup

                            Starting background process MARK

                            Mon Jul 18 12:16:24 2016

                            MARK started with pid=28, OS id=63968

                            NOTE: MARK has subscribed

                            lmon registered with NM - instance number 1 (internal mem no 0)

                            Reconfiguration started (old inc 0, new inc 2)

                            List of instances:

                            1 (myinst: 1)

                            Global Resource Directory frozen

                            * allocate domain 0, invalid = TRUE

                            Communication channels reestablished

                            Master broadcasted resource hash value bitmaps

                            Non-local Process blocks cleaned out

                            LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived

                            Set master node info

                            Submitted all remote-enqueue requests

                            Dwn-cvts replayed, VALBLKs dubious

                            All grantable enqueues granted

                            Post SMON to start 1st pass IR

                            Submitted all GCS remote-cache requests

                            Post SMON to start 1st pass IR

                            Fix write in gcs resources

                            Reconfiguration complete

                            Mon Jul 18 12:16:24 2016

                            LCK0 started with pid=30, OS id=63980

                            Starting background process RSMN

                            Mon Jul 18 12:16:24 2016

                            RSMN started with pid=31, OS id=63984

                            ORACLE_BASE from environment = /u01/app/oracle

                            Mon Jul 18 12:16:25 2016

                            ALTER DATABASE   MOUNT

                            This instance was first to mount

                            Mon Jul 18 12:16:26 2016

                            ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=172.16.0.217)(PORT=1521))' SCOPE=MEMORY SID='lnkdbee1';

                            NOTE: Loaded library: System

                            SUCCESS: diskgroup DATA was mounted

                            NOTE: dependency between database LNKDBEE and diskgroup resource ora.DATA.dg is established

                            Successful mount of redo thread 1, with mount id 3507955801

                            Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

                            Lost write protection disabled

                            Completed: ALTER DATABASE   MOUNT

                            Mon Jul 18 12:16:31 2016

                            ALTER DATABASE OPEN

                            This instance was first to open

                            Beginning crash recovery of 1 threads

                            parallel recovery started with 2 processes

                            Started redo scan

                            Completed redo scan

                            read 25 KB redo, 24 data blocks need recovery

                            Started redo application at

                            Thread 2: logseq 4919, block 9284, scn 302561480

                            Recovery of Online Redo Log: Thread 2 Group 5 Seq 4919 Reading mem 0

                              Mem# 0: +DATA/lnkdbee/onlinelog/group_5.521.859897543

                              Mem# 1: +DATA/lnkdbee/onlinelog/group_5.322.859897543

                            Completed redo application of 0.02MB

                            Completed crash recovery at

                            Thread 2: logseq 4919, block 9334, scn 302581541

                            24 data blocks read, 24 data blocks written, 25 redo k-bytes read

                            ARCH: STARTING ARCH PROCESSES

                            Mon Jul 18 12:16:32 2016

                            ARC0 started with pid=36, OS id=64052

                            ARC0: Archival started

                            ARCH: STARTING ARCH PROCESSES COMPLETE

                            ARC0: STARTING ARCH PROCESSES

                            Mon Jul 18 12:16:32 2016

                            ARC1 started with pid=37, OS id=64056

                            Mon Jul 18 12:16:32 2016

                            ARC2 started with pid=38, OS id=64060

                            Mon Jul 18 12:16:32 2016

                            ARC3 started with pid=39, OS id=64064

                            ARC1: Archival started

                            ARC2: Archival started

                            ARC1: Becoming the 'no FAL' ARCH

                            ARC1: Becoming the 'no SRL' ARCH

                            ARC2: Becoming the heartbeat ARCH

                            ARC3: Archival started

                            ARC0: STARTING ARCH PROCESSES COMPLETE

                            Thread 2 advanced to log sequence 4920 (thread recovery)

                            Picked broadcast on commit scheme to generate SCNs

                            Thread 2 advanced to log sequence 4921 (before internal thread enable)

                            Archived Log entry 9964 added for thread 2 sequence 4919 ID 0xcda746ed dest 1:

                            Archived Log entry 9965 added for thread 2 sequence 4920 ID 0xcda746ed dest 1:

                            Thread 2 advanced to log sequence 4922 (after internal thread enable)

                            Thread 1 opened at log sequence 5046

                              Current log# 3 seq# 5046 mem# 0: +DATA/lnkdbee/onlinelog/group_3.442.859897527

                              Current log# 3 seq# 5046 mem# 1: +DATA/lnkdbee/onlinelog/group_3.452.859897527

                            Successful open of redo thread 1

                            MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

                            SMON: enabling cache recovery

                            Redo thread 2 internally disabled at seq 4922 (CKPT)

                            Archived Log entry 9966 added for thread 2 sequence 4921 ID 0xcda746ed dest 1:

                            ARC0: Archiving disabled thread 2 sequence 4922

                            Archived Log entry 9967 added for thread 2 sequence 4922 ID 0xcda746ed dest 1:

                            [64038] Successfully onlined Undo Tablespace 2.

                            Undo initialization finished serial:0 start:4041858526 end:4041859036 diff:510 (5 seconds)

                            Verifying file header compatibility for 11g tablespace encryption..

                            Verifying 11g file header compatibility for tablespace encryption completed

                            SMON: enabling tx recovery

                            Database Characterset is WE8MSWIN1252

                            No Resource Manager plan active

                            Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee1/trace/lnkdbee1_ora_64038.trc:

                            ORA-00604: error occurred at recursive SQL level 1

                            ORA-00942: table or view does not exist

                            Errors in file /u01/app/oracle/diag/rdbms/lnkdbee/lnkdbee1/trace/lnkdbee1_ora_64038.trc:

                            ORA-00604: error occurred at recursive SQL level 1

                            ORA-00942: table or view does not exist

                            Error 604 happened during db open, shutting down database

                            USER (ospid: 64038): terminating the instance due to error 604

                            Instance terminated by USER, pid = 64038

                            ORA-1092 signalled during: ALTER DATABASE OPEN...

                            opiodr aborting process unknown ospid (64038) as a result of ORA-1092

                            Mon Jul 18 12:16:36 2016

                            ORA-1092 : opitsk aborting process

                            • 13. Re: RAC database will not open after enabling DB Vault
                              toonie

                              Hi,

                              ORA-00604: error occurred at recursive SQL level 1

                              ORA-00942: table or view does not exist

                              The actual table/view may not be represented in the alert log.

                               

                              You'll have to grep through the udump/bdump directory trace files for a trace file corresponding to the event.

                               

                              Regards,

                              Ruan

                              1 person found this helpful
                              • 14. Re: RAC database will not open after enabling DB Vault
                                Junaidullah.Khan-Oracle

                                Is the information in the provided trace incomplete? I ran the traces as suggested:


                                SQL> alter system set events '942  trace name ERRORSTACK level 3';
                                or
                                SQL> ALTER SYSTEM SET events = '942 TRACE NAME ERRORSTACK LEVEL 3';

                                 

                                Can you clarify on how to go about getting the proper trace, i.e. "You'll have to grep through the udump/bdump directory trace files for a trace file corresponding to the event."

                                 

                                Would really appreciate it

                                1 2 Previous Next