3 Replies Latest reply on Jun 6, 2018 2:36 AM by Beauty_and_dBest

    EBS 12.1.3  database archive logs apply hung?

    Beauty_and_dBest

      EBS 12.1.3

      11gR1

      OL6

       

       

      Hi ALL,

       

      I am restoring RMAN backup to our DR server.

      The restoration part is faster due to parameter > CONFIGURE DEVICE TYPE DISK PARALLELISM 8 BACKUP TYPE TO BACKUPSET;

      But when I started recovery part which is to apply achivelogs, it took very long time and seem hung, and the server shows no CPU activity.

       

      I checked the alert log, and I find below error:

       

      Full restore complete of datafile 11 /u01/oracle/PROD/db/apps_st/data/sysaux01.dbf.  Elapsed time: 0:13:59

        checkpoint is 5967219069127

      Mon Jun 04 18:03:00 2018

      alter database recover datafile list clear

      Completed: alter database recover datafile list clear

      alter database recover datafile list

      1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20

      Completed: alter database recover datafile list

      1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20

      alter database recover datafile list

      21 , 22 , 288 , 295 , 314 , 351 , 352 , 353 , 354 , 379 , 392 , 393 , 394 , 395 , 396 , 397 , 398 , 399 , 400 , 401

      Completed: alter database recover datafile list

      21 , 22 , 288 , 295 , 314 , 351 , 352 , 353 , 354 , 379 , 392 , 393 , 394 , 395 , 396 , 397 , 398 , 399 , 400 , 401

      alter database recover datafile list

      402 , 403 , 404 , 405 , 406 , 407

      Completed: alter database recover datafile list

      402 , 403 , 404 , 405 , 406 , 407

      alter database recover if needed

      start until cancel using backup controlfile

      Media Recovery Start

      Fast Parallel Media Recovery NOT enabled

      parallel recovery started with 3 processes

      ORA-279 signalled during: alter database recover if needed

      start until cancel using backup controlfile

      ...

      alter database recover logfile '/u02/ARCH/1_1798_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1798_965414376.dbf

      Mon Jun 04 18:06:25 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1798_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1799_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1799_965414376.dbf

      Mon Jun 04 18:12:49 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1799_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1800_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1800_965414376.dbf

      Mon Jun 04 18:18:13 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1800_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1801_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1801_965414376.dbf

      Mon Jun 04 18:23:36 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1801_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1802_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1802_965414376.dbf

      Mon Jun 04 18:43:47 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1802_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1803_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1803_965414376.dbf

      Mon Jun 04 19:56:21 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1803_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1804_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1804_965414376.dbf

      Mon Jun 04 21:07:36 2018

      Errors in file /u01/oracle/PROD/db/tech_st/11.1.0/admin/PROD_test01/diag/rdbms/prod/PROD/trace/PROD_ora_10806.trc  (incident=96129):

      ORA-00600: internal error code, arguments: [kcfidf_1], [1], [16], [kcvfdb.c], [1763], [], [], [], [], [], [], []

      Incident details in: /u01/oracle/PROD/db/tech_st/11.1.0/admin/PROD_test01/diag/rdbms/prod/PROD/incident/incdir_96129/PROD_ora_10806_i96129.trc

      Mon Jun 04 21:07:38 2018

      Trace dumping is performing id=[cdmp_20180604210738]

      Mon Jun 04 21:07:39 2018

      Sweep Incident[96129]: completed

      Mon Jun 04 21:10:53 2018

      ORA-279 signalled during: alter database recover logfile '/u02/ARCH/1_1804_965414376.dbf'...

      alter database recover logfile '/u02/ARCH/1_1805_965414376.dbf'

      Media Recovery Log /u02/ARCH/1_1805_965414376.dbf

      Mon Jun 04 21:17:11 2018

      Errors in file /u01/oracle/PROD/db/tech_st/11.1.0/admin/PROD_test01/diag/rdbms/prod/PROD/trace/PROD_ora_11242.trc  (incident=96137):

      ORA-00600: internal error code, arguments: [kcfidf_1], [1], [17], [kcvfdb.c], [1763], [], [], [], [], [], [], []

      Incident details in: /u01/oracle/PROD/db/tech_st/11.1.0/admin/PROD_test01/diag/rdbms/prod/PROD/incident/incdir_96137/PROD_ora_11242_i96137.trc

      Mon Jun 04 21:17:12 2018

      Trace dumping is performing id=[cdmp_20180604211712]

      Mon Jun 04 21:17:15 2018

      Sweep Incident[96137]: completed

       

       

      Please help how to troubleshoot this hung archive logs apply?

      Can I abort it and rerun the recovery?

       

       

      Kind regards,

      jc

        • 1. Re: EBS 12.1.3  database archive logs apply hung?
          Beauty_and_dBest

          Hi ALL,

           

          From the RMAN recovery screen run, I aborted it using CRTL+C

           

           

          RMAN> recover database;

           

          Starting recover at 04-JUN-18

          using channel ORA_DISK_1

          using channel ORA_DISK_2

          using channel ORA_DISK_3

          using channel ORA_DISK_4

          using channel ORA_DISK_5

          using channel ORA_DISK_6

          using channel ORA_DISK_7

          using channel ORA_DISK_8

           

          starting media recovery

           

          archived log file name=/u02/ARCH/1_1798_965414376.dbf thread=1 sequence=1798

          archived log file name=/u02/ARCH/1_1799_965414376.dbf thread=1 sequence=1799

           

          archived log file name=/u02/ARCH/1_1800_965414376.dbf thread=1 sequence=1800

          archived log file name=/u02/ARCH/1_1801_965414376.dbf thread=1 sequence=1801

          archived log file name=/u02/ARCH/1_1802_965414376.dbf thread=1 sequence=1802

          archived log file name=/u02/ARCH/1_1803_965414376.dbf thread=1 sequence=1803

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         archived log file name=/u02/ARCH/1_1804_965414376.dbf thread=1 sequence=1804

          archived log file name=/u02/ARCH/1_1805_965414376.dbf thread=1 sequence=1805

          ^C

          user interrupt received

          ^C

          user interrupt received

          RMAN-00571: ===========================================================

          RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

          RMAN-00571: ===========================================================

          RMAN-03002: failure of recover command at 06/04/2018 21:33:27

          ORA-00283: recovery session canceled due to errors

          RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/u02/ARCH/1_1805_965414376.dbf'

          ORA-00283: recovery session canceled due to errors

          ORA-01013: user requested cancel of current operation

           

          RMAN>

          • 2. Re: EBS 12.1.3  database archive logs apply hung?
            Maynard with Xenemm

            Restart your database into mount mode and try again. It applied some of the archivelogs, so there shouldn't be a problem to continue applying more. The ora-279 is normal for applying logs. The ora-600 errors look like they came from when you cancelled the process.

             

            Are the archivelogs available to the system or are they still in rman backup files?

             

            If they're available to the system, you could try applying them via sqlplus instead of rman, that gives you granular control to specify the archive log/redo log to apply and you can apply them one by one (or use auto)...