1 2 3 Previous Next 44 Replies Latest reply: Sep 22, 2012 10:55 PM by 918424 RSS

    Archive Logs NOT APPLIED but transferred

    918424
      Hi Gurus,
      I have configured Primary & Standby databases in same Oracle Home. OS version is OEL 5. Database version is 10.2.0.1. I could get the archive logs in the standby site but they are not getting applied in the standby database. I don't have OLAP installed in my database version. Would this create this issue? However I attached my primary alert log details below for your reference:

      Thu Aug 30 23:55:37 2012
      Starting ORACLE instance (normal)
      Cannot determine all dependent dynamic libraries for /proc/self/exe
      Unable to find dynamic library libocr10.so in search paths
      RPATH = /ade/aime1_build2101/oracle/has/lib/:/ade/aime1_build2101/oracle/lib/:/ade/aime1_build2101/oracle/has/lib/:
      LD_LIBRARY_PATH is not set!
      The default library directories are /lib and /usr/lib
      Unable to find dynamic library libocrb10.so in search paths
      Unable to find dynamic library libocrutl10.so in search paths
      Unable to find dynamic library libocrutl10.so in search paths
      LICENSE_MAX_SESSION = 0
      LICENSE_SESSIONS_WARNING = 0
      Picked latch-free SCN scheme 2
      Autotune of undo retention is turned on.
      IMODE=BR
      ILAT =18
      LICENSE_MAX_USERS = 0
      SYS auditing is disabled
      ksdpec: called for event 13740 prior to event group initialization
      Starting up ORACLE RDBMS Version: 10.2.0.1.0.
      System parameters with non-default values:
      processes = 150
      sga_target = 289406976
      control_files = /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim/control01.ctl, /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim/control02.ctl, /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim/control03.ctl
      db_file_name_convert = /home/oracle/oracle/product/10.2.0/db_1/oradata/newstand, /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim
      log_file_name_convert = /home/oracle/oracle/product/10.2.0/db_1/oradata/newstand, /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim, /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/NEWSTAND/onlinelog, /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/NEWPRIM/onlinelog
      db_block_size = 8192
      compatible = 10.2.0.1.0
      log_archive_config = DG_CONFIG=(newprim,newstand)
      log_archive_dest_1 = LOCATION=/home/oracle/oracle/product/10.2.0/db_1/oradata/newprim/arch/
      VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
      DB_UNIQUE_NAME=newprim
      log_archive_dest_2 = SERVICE=newstand LGWR ASYNC VALID_FOR=(online_logfiles,primary_role) DB_UNIQUE_NAME=newstand
      log_archive_dest_state_1 = enable
      log_archive_dest_state_2 = enable
      log_archive_max_processes= 30
      log_archive_format = %t_%s_%r.dbf
      fal_client = newprim
      fal_server = newstand
      db_file_multiblock_read_count= 16
      db_recovery_file_dest = /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area
      db_recovery_file_dest_size= 2147483648
      standby_file_management = AUTO
      undo_management = AUTO
      undo_tablespace = UNDOTBS1
      remote_login_passwordfile= EXCLUSIVE
      db_domain =
      dispatchers = (PROTOCOL=TCP) (SERVICE=newprimXDB)
      job_queue_processes = 10
      background_dump_dest = /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump
      user_dump_dest = /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/udump
      core_dump_dest = /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/cdump
      audit_file_dest = /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/adump
      db_name = newprim
      db_unique_name = newprim
      open_cursors = 300
      pga_aggregate_target = 95420416
      PMON started with pid=2, OS id=28091
      PSP0 started with pid=3, OS id=28093
      MMAN started with pid=4, OS id=28095
      DBW0 started with pid=5, OS id=28097
      LGWR started with pid=6, OS id=28100
      CKPT started with pid=7, OS id=28102
      SMON started with pid=8, OS id=28104
      RECO started with pid=9, OS id=28106
      CJQ0 started with pid=10, OS id=28108
      MMON started with pid=11, OS id=28110
      MMNL started with pid=12, OS id=28112
      Thu Aug 30 23:55:38 2012
      starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
      starting up 1 shared server(s) ...
      Thu Aug 30 23:55:38 2012
      ALTER DATABASE MOUNT
      Thu Aug 30 23:55:42 2012
      Setting recovery target incarnation to 2
      Thu Aug 30 23:55:43 2012
      Successful mount of redo thread 1, with mount id 1090395834
      Thu Aug 30 23:55:43 2012
      Database mounted in Exclusive Mode
      Completed: ALTER DATABASE MOUNT
      Thu Aug 30 23:55:43 2012
      ALTER DATABASE OPEN
      Thu Aug 30 23:55:43 2012
      LGWR: STARTING ARCH PROCESSES
      ARC0 started with pid=16, OS id=28122
      ARC1 started with pid=17, OS id=28124
      ARC2 started with pid=18, OS id=28126
      ARC3 started with pid=19, OS id=28128
      ARC4 started with pid=20, OS id=28133
      ARC5 started with pid=21, OS id=28135
      ARC6 started with pid=22, OS id=28137
      ARC7 started with pid=23, OS id=28139
      ARC8 started with pid=24, OS id=28141
      ARC9 started with pid=25, OS id=28143
      ARCa started with pid=26, OS id=28145
      ARCb started with pid=27, OS id=28147
      ARCc started with pid=28, OS id=28149
      ARCd started with pid=29, OS id=28151
      ARCe started with pid=30, OS id=28153
      ARCf started with pid=31, OS id=28155
      ARCg started with pid=32, OS id=28157
      ARCh started with pid=33, OS id=28159
      ARCi started with pid=34, OS id=28161
      ARCj started with pid=35, OS id=28163
      ARCk started with pid=36, OS id=28165
      ARCl started with pid=37, OS id=28167
      ARCm started with pid=38, OS id=28169
      ARCn started with pid=39, OS id=28171
      ARCo started with pid=40, OS id=28173
      ARCp started with pid=41, OS id=28175
      ARCq started with pid=42, OS id=28177
      ARCr started with pid=43, OS id=28179
      ARCs started with pid=44, OS id=28181
      Thu Aug 30 23:55:44 2012
      ARC0: Archival started
      ARC1: Archival started
      ARC2: Archival started
      ARC3: Archival started
      ARC4: Archival started
      ARC5: Archival started
      ARC6: Archival started
      ARC7: Archival started
      ARC8: Archival started
      ARC9: Archival started
      ARCa: Archival started
      ARCb: Archival started
      ARCc: Archival started
      ARCd: Archival started
      ARCe: Archival started
      ARCf: Archival started
      ARCg: Archival started
      ARCh: Archival started
      ARCi: Archival started
      ARCj: Archival started
      ARCk: Archival started
      ARCl: Archival started
      ARCm: Archival started
      ARCn: Archival started
      ARCo: Archival started
      ARCp: Archival started
      ARCq: Archival started
      ARCr: Archival started
      ARCs: Archival started
      ARCt: Archival started
      LGWR: STARTING ARCH PROCESSES COMPLETE
      ARCt started with pid=45, OS id=28183
      LNS1 started with pid=46, OS id=28185
      Thu Aug 30 23:55:48 2012
      Thread 1 advanced to log sequence 68
      Thu Aug 30 23:55:48 2012
      ARCo: Becoming the 'no FAL' ARCH
      ARCo: Becoming the 'no SRL' ARCH
      Thu Aug 30 23:55:48 2012
      ARCp: Becoming the heartbeat ARCH
      Thu Aug 30 23:55:48 2012
      Thread 1 opened at log sequence 68
      Current log# 1 seq# 68 mem# 0: /home/oracle/oracle/product/10.2.0/db_1/oradata/newprim/redo01.log
      Successful open of redo thread 1
      Thu Aug 30 23:55:48 2012
      MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
      Thu Aug 30 23:55:48 2012
      SMON: enabling cache recovery
      Thu Aug 30 23:55:48 2012
      Successfully onlined Undo Tablespace 1.
      Thu Aug 30 23:55:48 2012
      SMON: enabling tx recovery
      Thu Aug 30 23:55:49 2012
      Database Characterset is WE8ISO8859P1
      replication_dependency_tracking turned off (no async multimaster replication found)
      Starting background process QMNC
      QMNC started with pid=47, OS id=28205
      Thu Aug 30 23:55:49 2012
      Error 1034 received logging on to the standby
      Thu Aug 30 23:55:49 2012
      Errors in file /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc:
      ORA-01034: ORACLE not available
      FAL[server, ARC1]: Error 1034 creating remote archivelog file 'newstand'
      FAL[server, ARC1]: FAL archive failed, see trace file.
      Thu Aug 30 23:55:49 2012
      Errors in file /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc:
      ORA-16055: FAL request rejected
      ARCH: FAL archive failed. Archiver continuing
      Thu Aug 30 23:55:49 2012
      ORACLE Instance newprim - Archival Error. Archiver continuing.
      Thu Aug 30 23:55:49 2012
      db_recovery_file_dest_size of 2048 MB is 9.77% used. This is a
      user-specified limit on the amount of space that will be used by this
      database for recovery-related files, and does not reflect the amount of
      space available in the underlying filesystem or ASM diskgroup.
      Thu Aug 30 23:55:50 2012
      Errors in file /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/udump/newprim_ora_28120.trc:
      ORA-00604: error occurred at recursive SQL level 1
      ORA-12663: Services required by client not available on the server
      ORA-36961: Oracle OLAP is not available.
      ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
      ORA-06512: at line 15
      Thu Aug 30 23:55:50 2012
      Completed: ALTER DATABASE OPEN
      Thu Aug 30 23:56:33 2012
      FAL[server]: Fail to queue the whole FAL gap
      GAP - thread 1 sequence 1-33
      DBID 1090398314 branch 792689455

      Kindly, guide me please..

      -Vimal.
        • 1. Re: Archive Logs NOT APPLIED but transferred
          Mahir M. Quluzade
          Hi,

          Your standby database is opened ? if oracle database 10g standby database is opened then, received archive logs is not appling.
          Then you must open standby database with mount mode and
          on standby
            SQL> shutdown immediate; 
            SQL> startup mount; 
            SQL> alter database recover managed standby database using current logfile disconnect from  session; --- starting Redo Apply Service MRP0
          Regards
          Mahir M. Quluzade
          • 2. Re: Archive Logs NOT APPLIED but transferred
            Mahir M. Quluzade
            Can you paste here results ?
             on primary 
             
            SQL> select max(sequence#)  from v$archived_log; 
             
             on standby 
             
            SQL> select max(sequence#)  from v$archived_log; 
            SQL> select max(sequence#)  from v$archived_log where APPLIED='YES'; 
            • 3. Re: Archive Logs NOT APPLIED but transferred
              918424
              Hi Mahir M. Quluzade, thanks for the reply...

              On PRIMARY, the result of select max(sequence#) from v$archived_log; returns 67.

              Same query on STANDBY returns 66.

              Whereas, when Applied='YES' on STANDBY, it returns Nothing...

              Thanks.
              • 4. Re: Archive Logs NOT APPLIED but transferred
                Hemant K Chitale
                Thu Aug 30 23:55:49 2012
                Error 1034 received logging on to the standby
                Thu Aug 30 23:55:49 2012
                Errors in file /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc:
                ORA-01034: ORACLE not available
                FAL[server, ARC1]: Error 1034 creating remote archivelog file 'newstand'
                FAL[server, ARC1]: FAL archive failed, see trace file.
                Thu Aug 30 23:55:49 2012
                Errors in file /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc:
                ORA-16055: FAL request rejected
                ARCH: FAL archive failed. Archiver continuing
                The Primary could not connect to the Standby.

                Hemant K Chitale
                • 5. Re: Archive Logs NOT APPLIED but transferred
                  Shivananda Rao
                  From Primary database, please post the outcome of this:
                  sql>select severity,error_code,message,to_char(timestamp,'DD-MON-YYYY HH24:MI:SS') from v$dataguard_status where dest_id=2;
                  I assume that log_archive_dest_2 is the parameter set on the primary database to ship logs to the standby database.
                  • 6. Re: Archive Logs NOT APPLIED but transferred
                    918424
                    Hi Shiva, kindly find below the query output:


                    SQL> select severity,error_code,message,to_char(timestamp,'DD-MON-YYYY HH24:MI:SS') from v$dataguard_status where dest_id=2;

                    SEVERITY ERROR_CODE MESSAGE TO_CHAR(TIMESTAMP,'D
                    --------------- ---------- ---------------------------------------------------------------------- --------------------
                    Error 1034 FAL[server, ARC1]: Error 1034 creating remote archivelog file 'newstan 30-AUG-2012 23:55:49
                    d'


                    Hemant: But I am getting the archive logs in the standby site...

                    -Vimal.
                    • 7. Re: Archive Logs NOT APPLIED but transferred
                      Mahir M. Quluzade
                      Did you added standby logs ?
                      • 8. Re: Archive Logs NOT APPLIED but transferred
                        Hemant K Chitale
                        Your are getting files now.

                        But there is gap when the files did not go through : Sequences 1 to 33
                        FAL[server]: Fail to queue the whole FAL gap
                        GAP - thread 1 sequence 1-33
                        (I wonder how you've setup a standby that is requesting even Sequence 1 !!!!)

                        Hemant K Chitale
                        • 9. Re: Archive Logs NOT APPLIED but transferred
                          Mahir M. Quluzade
                          I think you must create standby again.

                          How to created standby database?
                          • 10. Re: Archive Logs NOT APPLIED but transferred
                            CKPT
                            Paste the trace content of trace file.

                            1) cat /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc


                            Option 2:(workaround)
                            As a workaround, You can create only new standby control file and restore on standby then start MRP see if any errors still.
                            • 11. Re: Archive Logs NOT APPLIED but transferred
                              918424
                              CKPT: The trace file details are added below for your reference;


                              /home/oracle/oracle/product/10.2.0/db_1/admin/newprim/bdump/newprim_arc1_28124.trc
                              Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
                              With the Partitioning and Data Mining options
                              ORACLE_HOME = /home/oracle/oracle/product/10.2.0/db_1
                              System name:     Linux
                              Node name:     localhost.localdomain
                              Release:     2.6.18-8.el5PAE
                              Version:     #1 SMP Tue Jun 5 23:39:57 EDT 2007
                              Machine:     i686
                              Instance name: newprim
                              Redo thread mounted by this instance: 1
                              Oracle process number: 17
                              Unix process pid: 28124, image: oracle@localhost.localdomain (ARC1)

                              *** SERVICE NAME:() 2012-08-30 23:55:48.314
                              *** SESSION ID:(155.1) 2012-08-30 23:55:48.314
                              kcrrwkx: nothing to do (start)
                              Redo shipping client performing standby login
                              OCISessionBegin failed -1
                              .. Detailed OCI error val is 1034 and errmsg is 'ORA-01034: ORACLE not available
                              '
                              *** 2012-08-30 23:55:49.723 60679 kcrr.c
                              Error 1034 received logging on to the standby
                              Error 1034 connecting to destination LOG_ARCHIVE_DEST_2 standby host 'newstand'
                              Error 1034 attaching to destination LOG_ARCHIVE_DEST_2 standby host 'newstand'
                              ORA-01034: ORACLE not available
                              *** 2012-08-30 23:55:49.723 58941 kcrr.c
                              kcrrfail: dest:2 err:1034 force:0 blast:1
                              kcrrwkx: unknown error:1034
                              ORA-16055: FAL request rejected
                              ARCH: Connecting to console port...
                              ARCH: Connecting to console port...
                              kcrrwkx: nothing to do (end)
                              *** 2012-08-31 00:00:43.417
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:05:43.348
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:10:43.280
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:15:43.217
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:20:43.160
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:25:43.092
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:30:43.031
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:35:42.961
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:40:42.890
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:45:42.820
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:50:42.755
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 00:55:42.686
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 01:00:42.631
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 01:05:42.565
                              kcrrwkx: nothing to do (start)
                              *** 2012-08-31 01:10:42.496
                              kcrrwkx: nothing to do (start)


                              Mahir: Yes I have my 4 standby redo logs!

                              I created the standby manually without using RMAN.

                              Hemant: if it asks for even first thread, then obviously it shows nothing is applied on Standby. By the way so it is not called a 'GAP', I think..!

                              Thanks.
                              • 12. Re: Archive Logs NOT APPLIED but transferred
                                mseberg
                                Hello;

                                ORA-01034, So the standby database has stopped running.

                                You have errors that are somewhat misleading.

                                I would shutdown ( just to be safe )

                                startup nomount;

                                alter database mount standby database ;

                                ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;


                                kcrrwkx: nothing to do - I think this is Bug 4883174 - fixed in 10.2.0.3



                                Best Regards

                                mseberg


                                h2. Later

                                Double check your password file on the standby database for a typo. If its incorrect fix it and restart the standby on the correct password file.

                                Would also double check these on both sides :

                                LOG_ARCHIVE_DEST_N
                                LOG_ARCHIVE_DEST_STATE_N

                                Make sure standby_archive_dest in NOT in either INIT file.

                                Edited by: mseberg on Sep 21, 2012 5:16 AM
                                • 13. Re: Archive Logs NOT APPLIED but transferred
                                  CKPT
                                  Hello,

                                  Need some more information.

                                  1) show parameter LOCAL_LISTENER
                                  2) show parameter REMOTE_LOGIN_
                                  Ensure it is shared or exclusive.

                                  3) show parameter STANDBY_ARCHIVE_DEST

                                  if not set, change it to as local archive destination


                                  I guess, it's an configuration issue and no need to recreate whole standby environment. Let's see

                                  Edited by: CKPT on Sep 21, 2012 3:44 PM
                                  • 14. Re: Archive Logs NOT APPLIED but transferred
                                    abdul:
                                    hello,

                                    can you check this parameter:

                                    What is the value of log_archive_dest_state_2

                                    Do the following
                                    [on primary]
                                    SQL>show parameter  log_archive_dest_state_2;
                                    B.O.L

                                    abdul

                                    Edited by: abdul_ora on Sep 21, 2012 3:43 PM
                                    1 2 3 Previous Next