14 Replies Latest reply: Apr 5, 2013 3:53 PM by Artem S. Tsygankov RSS

    logical standby: ORA-01031: insufficient privileges

    Artem S. Tsygankov
      Dear Colleagues,
      Today in my Logical Standby don't apply archivelogs and I see an error in alert.log:

      ORA-26808: Apply process AS01 died unexpectedly.
      ORA-01031: insufficient privileges

      Also I see next logs in trace files:
      h4.
      1)
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Data Mining and Real Application Testing options
      ORACLE_HOME = /ora/Ora11203
      System name: Linux
      Node name: base
      Release: 2.6.18-308.20.1.el5
      Version: #1 SMP Tue Nov 6 04:38:29 EST 2012
      Machine: x86_64
      Instance name: oracle
      Redo thread mounted by this instance: 1
      Oracle process number: 43
      Unix process pid: 16634, image: oracle@base (AS01)


      *** 2013-04-04 16:58:47.062
      *** SESSION ID:(146.16811) 2013-04-04 16:58:47.062
      *** CLIENT ID:() 2013-04-04 16:58:47.062
      *** SERVICE NAME:(SYS$USERS) 2013-04-04 16:58:47.062
      *** MODULE NAME:(Streams) 2013-04-04 16:58:47.062
      *** ACTION NAME:( - Apply Server) 2013-04-04 16:58:47.062

      knasplcr: eager error was not rolled back
      ++ LCR Dump Begin: 0x2b0cf2a6d168 - ddl
      op: 5, Original op: 5, baseobjn: 0, objn: 0, objv: 0
      DF: 0x00000002, DF2: 0x00000010, MF: 0x00020810, MF2: 0x00000000
      PF: 0x00000000, PF2: 0x08000000
      MergeFlag: 0x00, FilterFlag: 0x00
      Id: 0, iotPrimaryKeyCount: 0, numChgRec: 1
      NumCrSpilled: 0
      RedoThread#: 1, rba: 0x0213d7.00023455.01ac
      scn: 0x0002.fc0f8769, (scn: 0x0000.00000000, scn_sqn: 0, lcr_sqn: 0)xid: 0x0019.00e.0007f8d9, parentxid: 0x0019.00e.0007f8d9, proxyxid: 0x0000.000.00000000
      ncol: 0 newcount: 24, oldcount: 0
      LUBA: 0x4.1000ad1.e.0.0
      ++ LCR Dump Begin: 0x59b1afdc8 - commit
      op: 7, Original op: 7, baseobjn: 0, objn: 0, objv: 0
      DF: 0x00000002, DF2: 0x00000010, MF: 0x00220000, MF2: 0x02000000
      PF: 0x00100000, PF2: 0x08040000
      MergeFlag: 0x03, FilterFlag: 0x00
      Id: 3, iotPrimaryKeyCount: 0, numChgRec: 0
      NumCrSpilled: 0
      RedoThread#: 1, rba: 0x0213d7.00023456.0108
      scn: 0x0002.fc0f876b, (scn: 0x0002.fc0f876b, scn_sqn: 1, lcr_sqn: 1)xid: 0x0019.00e.0007f8d9, parentxid: 0x0019.00e.0007f8d9, proxyxid: 0x0000.000.00000000
      ncol: 0 newcount: 0, oldcount: 0
      LUBA: 0x4.1000ad1.e.0.0
      Apply Slave is exiting due to error ORA-1031KSV 1031 error in slave process

      *** 2013-04-04 16:58:47.067
      ORA-01031: insufficient privileges
      OPIRIP: Uncaught error 447. Error stack:
      ORA-00447: fatal error in background process
      ORA-01031: insufficient privileges
      h4.
      2)
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Data Mining and Real Application Testing options
      ORACLE_HOME = /ora/Ora11203
      System name: Linux
      Node name: base
      Release: 2.6.18-308.20.1.el5
      Version: #1 SMP Tue Nov 6 04:38:29 EST 2012
      Machine: x86_64
      Instance name: oracle
      Redo thread mounted by this instance: 1
      Oracle process number: 30
      Unix process pid: 16404, image: oracle@base (LSP0)


      *** 2013-04-04 16:41:58.401
      *** SESSION ID:(294.35251) 2013-04-04 16:41:58.401
      *** CLIENT ID:() 2013-04-04 16:41:58.401
      *** SERVICE NAME:(SYS$BACKGROUND) 2013-04-04 16:41:58.401
      *** MODULE NAME:() 2013-04-04 16:41:58.401
      *** ACTION NAME:() 2013-04-04 16:41:58.401

      knahcapplymain: encountered error=26808

      *** 2013-04-04 16:41:58.401
      dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
      ----- Error Stack Dump -----
      ORA-26808: Apply process AS02 died unexpectedly.
      ORA-01031: insufficient privileges
      KNACDMP: *******************************************************
      KNACDMP: Dumping apply coordinator's context at 25487cb0
      KNACDMP: Apply Engine # 0
      KNACDMP: Apply Engine name
      KNACDMP: Coordinator's Watermarks ------------------------------
      KNACDMP: Apply High Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Apply Low Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Recovery Low Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Fetch Low Watermark = 0002fc0f876b (SCN=0x0002.fc0f876b)
      KNACDMP: Fetch Low Watermark Time = 811857327
      KNACDMP: Oldest SCN = (SCN=0x0000.00000000)
      KNACDMP: Oldest XID =
      KNACDMP: Oldest Create Time = 0
      KNACDMP: Last replicant syncpoint SCN = 0x0000.00000000
      KNACDMP: Last syncpoint at primary SCN = 0x0002.fc0f875f
      KNACDMP: First partition max pos = 0002fc10fab4 (SCN=0x0002.fc10fab4)
      KNACDMP: Last partition max pos = 0002fc10fab4 (SCN=0x0002.fc10fab4)
      KNACDMP: Last processed = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Conservative pos = 0002fc0f8cb2 (SCN=0x0002.fc0f8cb2)
      KNACDMP: Recovery start pos = (SCN=0x0000.00000000)
      KNACDMP: Recovery high watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Spill LWM = (SCN=0x0000.00000000)
      KNACDMP: Spill LWM Create Time = 0
      KNACDMP: Coordinator's constants -------------------------------
      KNACDMP: number of apply slaves = 5
      KNACDMP: min number of apply slaves = 5
      KNACDMP: max number of apply slaves = 5
      KNACDMP: safety level (K) = 1
      KNACDMP: max txns in memory = 400
      KNACDMP: max constraints per table = 620
      KNACDMP: hash table size (in entries) = 10000000
      KNACDMP: Coordinator's intervals -------------------------------
      KNACDMP: syncpoint interval (ms) = 0
      KNACDMP: write low watermark interval(ms)= 1
      KNACDMP: Coordinator's timers/counters -------------------------
      KNACDMP: current time = 1365082918
      KNACDMP: low watermark timer = 0
      KNACDMP: syncpoint timer = 1365082918
      KNACDMP: txnbufsize timer = 1365082220
      KNACDMP: Coordinator's txn counts -------------------------
      KNACDMP: total txns applied = 0
      KNACDMP: number of unassigned comp txns = 0
      KNACDMP: number of unassigned incomp txns= 0
      KNACDMP: avg number of unassigned txns = 0.00
      KNACDMP: total applied at last plwm write= 0
      KNACDMP: apply prog. entries below plwm = 0
      KNACDMP: total unassigned lcrs = 0
      KNACDMP: Coordinator's State/Flags -----------------------------
      KNACDMP: Coordinator's State = KNACST_APPLY_UNTIL_END
      KNACDMP: Coordinator's Flags = 0x408004
      KNACDMP: Slave counts ------------------------------------------
      KNACDMP: number of reserved slaves = 0
      KNACDMP: number of admin slaves = 0
      KNACDMP: number of slaves in wait cmt = 1
      KNACDMP: number of slaves suspended = 0
      KNACDMP: number of safe slaves = 1
      KNACDMP: avg number of idle slaves = 0.00
      KNACDMP: number of slaves initializing = 0
      KNACDMP: number of slaves terminating = 0
      KNACDMP: Slave Lists -------------------------------------------
      KNACDMP: Dumping All Slaves :-
      Slave id = 0, State = 8, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 1, State = 9, Flags = 2, Assigned Xid = 0x001e.006.0005d213 1 txns 0 lcrs
      Slave id = 2, State = 5, Flags = 1, Assigned Xid = 0x0019.00e.0007f8d9 1 txns 0 lcrs
      Slave id = 3, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 4, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 5, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      KNACDMP: End dumping all slaves
      KNACDMP: syncdep slaves = { }
      KNACDMP: cont chunk slaves = { }
      KNACDMP: cont slaves = { }
      KNACDMP: exec txn slaves = { }
      KNACDMP: Active slaves (2) = { 1 2 }
      KNACDMP: Idle slaves (3) = { 3 4 5 }
      KNACDMP: Txn Lists ---------------------------------------------
      KNACDMP: Dumping all txns :-
      XID = 0x001e.006.0005d213 Commit pos = 0002fc06718c (SCN=0x0002.fc06718c) State = 0
      Lcr cnt = 0
      Assigned to slavid = 1
      Fetched chunks = 142
      depslaves = { } wm depslaves = { }
      ---
      XID = 0x0019.00e.0007f8d9 Commit pos = 0002fc0f876b (SCN=0x0002.fc0f876b) State = 1
      Lcr cnt = 0
      Assigned to slavid = 2
      Fetched chunks = 1
      depslaves = { } wm depslaves = { }
      ---
      KNACDMP: End dumping all txns.
      KNACDMP: Complete txns = { 0x0019.00e.0007f8d9 ** NO UNASS ** }
      KNACDMP: Unassigned txns = { }
      KNACDMP: *******************************************************
      Warning: Apply error received: ORA-26714: User Error encountered during apply process. Clearing.
      knahcapplymain: encountered error=26808

      *** 2013-04-04 16:58:47.073
      dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
      ----- Error Stack Dump -----
      ORA-26808: Apply process AS01 died unexpectedly.
      ORA-01031: insufficient privileges
      KNACDMP: *******************************************************
      KNACDMP: Dumping apply coordinator's context at 25487cb0
      KNACDMP: Apply Engine # 0
      KNACDMP: Apply Engine name
      KNACDMP: Coordinator's Watermarks ------------------------------
      KNACDMP: Apply High Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Apply Low Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Recovery Low Watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Fetch Low Watermark = 0002fc0f876b (SCN=0x0002.fc0f876b)
      KNACDMP: Fetch Low Watermark Time = 811857327
      KNACDMP: Oldest SCN = (SCN=0x0000.00000000)
      KNACDMP: Oldest XID =
      KNACDMP: Oldest Create Time = 0
      KNACDMP: Last replicant syncpoint SCN = 0x0000.00000000
      KNACDMP: Last syncpoint at primary SCN = 0x0002.fc0f875f
      KNACDMP: First partition max pos = 0002fc10fab4 (SCN=0x0002.fc10fab4)
      KNACDMP: Last partition max pos = 0002fc10fab4 (SCN=0x0002.fc10fab4)
      KNACDMP: Last processed = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Conservative pos = 0002fc0f8cb2 (SCN=0x0002.fc0f8cb2)
      KNACDMP: Recovery start pos = (SCN=0x0000.00000000)
      KNACDMP: Recovery high watermark = 0002fc0f875f (SCN=0x0002.fc0f875f)
      KNACDMP: Spill LWM = (SCN=0x0000.00000000)
      KNACDMP: Spill LWM Create Time = 0
      KNACDMP: Coordinator's constants -------------------------------
      KNACDMP: number of apply slaves = 5
      KNACDMP: min number of apply slaves = 5
      KNACDMP: max number of apply slaves = 5
      KNACDMP: safety level (K) = 1
      KNACDMP: max txns in memory = 400
      KNACDMP: max constraints per table = 620
      KNACDMP: hash table size (in entries) = 10000000
      KNACDMP: Coordinator's intervals -------------------------------
      KNACDMP: syncpoint interval (ms) = 0
      KNACDMP: write low watermark interval(ms)= 1
      KNACDMP: Coordinator's timers/counters -------------------------
      KNACDMP: current time = 1365083926
      KNACDMP: low watermark timer = 0
      KNACDMP: syncpoint timer = 1365083926
      KNACDMP: txnbufsize timer = 1365083218
      KNACDMP: Coordinator's txn counts -------------------------
      KNACDMP: total txns applied = 0
      KNACDMP: number of unassigned comp txns = 0
      KNACDMP: number of unassigned incomp txns= 1
      KNACDMP: avg number of unassigned txns = 0.00
      KNACDMP: total applied at last plwm write= 0
      KNACDMP: apply prog. entries below plwm = 0
      KNACDMP: total unassigned lcrs = 0
      KNACDMP: Coordinator's State/Flags -----------------------------
      KNACDMP: Coordinator's State = KNACST_APPLY_UNTIL_END
      KNACDMP: Coordinator's Flags = 0x8204
      KNACDMP: Slave counts ------------------------------------------
      KNACDMP: number of reserved slaves = 0
      KNACDMP: number of admin slaves = 0
      KNACDMP: number of slaves in wait cmt = 0
      KNACDMP: number of slaves suspended = 0
      KNACDMP: number of safe slaves = 1
      KNACDMP: avg number of idle slaves = 0.00
      KNACDMP: number of slaves initializing = 0
      KNACDMP: number of slaves terminating = 0
      KNACDMP: Slave Lists -------------------------------------------
      KNACDMP: Dumping All Slaves :-
      Slave id = 0, State = 8, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 1, State = 5, Flags = 1, Assigned Xid = 0x0019.00e.0007f8d9 1 txns 0 lcrs
      Slave id = 2, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 3, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 4, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      Slave id = 5, State = 0, Flags = 0, Not Assigned 0 txns 0 lcrs
      KNACDMP: End dumping all slaves
      KNACDMP: syncdep slaves = { }
      KNACDMP: cont chunk slaves = { }
      KNACDMP: cont slaves = { }
      KNACDMP: exec txn slaves = { }
      KNACDMP: Active slaves (1) = { 1 }
      KNACDMP: Idle slaves (4) = { 2 3 4 5 }
      KNACDMP: Txn Lists ---------------------------------------------
      KNACDMP: Dumping all txns :-
      XID = 0x001e.006.0005d213 Commit pos = 0002fc06718c (SCN=0x0002.fc06718c) State = 0
      Lcr cnt = 0
      Not Assigned
      Fetched chunks = 142
      depslaves = { } wm depslaves = { }
      ---
      XID = 0x0019.00e.0007f8d9 Commit pos = 0002fc0f876b (SCN=0x0002.fc0f876b) State = 1
      Lcr cnt = 0
      Assigned to slavid = 1
      Fetched chunks = 1
      depslaves = { } wm depslaves = { }
      ---
      KNACDMP: End dumping all txns.
      KNACDMP: Complete txns = { 0x0019.00e.0007f8d9 ** NO UNASS ** }
      KNACDMP: Unassigned txns = { 0x001e.006.0005d213 }
      KNACDMP: *******************************************************

      *** 2013-04-04 16:58:47.513
      Warning: Apply error received: ORA-26714: User Error encountered during apply process. Clearing.

      I watched this link http://docs.oracle.com/cd/E11882_01/server.112/e17069/strms_trapply.htm#i1014714 and I checked grants of schemas owners. These grants identical as on Primary DB Server.
      I don't know what I need to do. It's very critical Server. Please help me.