8 Replies Latest reply: Oct 25, 2013 3:08 AM by TzuWen, Lin RSS

    Logs are not shipped to the DR database. Please help

    TzuWen, Lin

      Hello ,

       

      I create a physical standby databas at local site. the dataguard works.

       

      But the standby database cannot receive log when the standby database moved to remote and change IP ,/etc/hosts ,listener.ora and tnsnames.ora.

       

      It seems like network problem coz log shipping works when standby database at local site.

       

      Please help.

       

      message below

      AIX 5300-12-04-1119 + Oracle 11.2.0.2

       

       

      ######### Primary database alert log. #########

      ******************************************************************

      LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2

      ******************************************************************

      Wed Oct 09 18:40:13 2013

      WARN: ARC1: Terminating pid 2916466 hung on an I/O operation

      WARN: ARC1: Terminating pid 3350692 hung on an I/O operation

      krsv_proc_kill: Killing 1 processes (Process by index)

      Wed Oct 09 18:40:24 2013

      krsv_proc_kill: Killing 1 processes (Process by index)

      ARC1: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2

      ARC1: Detected ARCH process failure

      ARC1: Detected ARCH process failure

      ARC1: STARTING ARCH PROCESSES

      Wed Oct 09 18:40:27 2013

      ARC2 started with pid=20, OS id=2916468

      ARC2: Archival started

      WARN: ARC2: Terminating pid 585944 hung on an I/O operation

      Wed Oct 09 18:40:27 2013

      ARC3 started with pid=22, OS id=3383458

      ARC3: Archival started

      ARC1: STARTING ARCH PROCESSES COMPLETE

      Reclaiming FAL entry from dead process [pid 2916466]

      krsv_proc_kill: Killing 1 processes (Process by index)

      WARN: ARC3: Terminating pid 585944 hung on an I/O operation

      krsv_proc_kill: Killing 1 processes (Process by index)

      ARC2: Detected ARCH process failure

      ARC2: STARTING ARCH PROCESSES

      Wed Oct 09 18:40:34 2013

      ARC0 started with pid=19, OS id=3854466

      ARC3: Becoming the heartbeat ARCH

      ARC0: Archival started

      ARC2: STARTING ARCH PROCESSES COMPLETE

      Reclaiming FAL entry from dead process [pid 3350692]

      Reclaiming FAL entry from dead process [pid 585944]

      Wed Oct 09 18:45:28 2013

      WARN: ARC1: Terminating pid 3854466 hung on an I/O operation

      WARN: ARC1: Terminating pid 2916468 hung on an I/O operation

      WARN: ARC1: Terminating pid 3383458 hung on an I/O operation

      Wed Oct 09 18:45:42 2013

      WARN: ARC1: Terminating pid 3858682 hung on an I/O operation

      krsv_proc_kill: Killing 1 processes (Process by index)

      krsv_proc_kill: Killing 1 processes (Process by index)

      krsv_proc_kill: Killing 1 processes (Process by index)

      Wed Oct 09 18:45:53 2013

      krsv_proc_kill: Killing 1 processes (Process by index)

      ARC1: Detected ARCH process failure

      ARC1: Detected ARCH process failure

      ARC1: Detected ARCH process failure

      ARC1: STARTING ARCH PROCESSES

      Wed Oct 09 18:45:55 2013

      ARC0 started with pid=19, OS id=3858686

      Wed Oct 09 18:45:55 2013

      ARC2 started with pid=20, OS id=3383460

      ARC0: Archival started

      Wed Oct 09 18:45:55 2013

      ARC3 started with pid=22, OS id=585962

      ARC2: Archival started

      ARC2: Becoming the heartbeat ARCH

      Reclaiming FAL entry from dead process [pid 3383458]

      ARC3: Archival started

      ARC1: STARTING ARCH PROCESSES COMPLETE

      Reclaiming FAL entry from dead process [pid 2916468]

      Wed Oct 09 18:46:57 2013

      Reclaiming FAL entry from dead process [pid 3854466]

      Wed Oct 09 18:46:59 2013

      NSA2 started with pid=47, OS id=2838532

      Wed Oct 09 18:47:02 2013

      Thread 1 advanced to log sequence 4883 (LGWR switch)

        Current log# 2 seq# 4883 mem# 0: /u2/oracle/oradata/plmdb/redo02.log

      Wed Oct 09 18:47:02 2013

      Archived Log entry 4860 added for thread 1 sequence 4882 ID 0x5c432f01 dest 1:

       

      ######### Standby database alert log. #########

      krsv_proc_kill: Killing 1 processes (idle RFS by thread/sequence)

      RFS[66]: Assigned to RFS process 700480

      RFS[66]: Opened log for thread 1 sequence 4872 dbid 1547947009 branch 757523841

      Wed Oct 09 18:46:59 2013

      Primary database is in MAXIMUM PERFORMANCE mode

      RFS[67]: Assigned to RFS process 463046

      RFS[67]: No standby redo logfiles available for thread 1

      RFS[67]: Opened log for thread 1 sequence 4883 dbid 1547947009 branch 757523841

      Wed Oct 09 18:51:03 2013

      RFS[64]: Possible network disconnect with primary database

      Wed Oct 09 18:51:15 2013

      krsv_proc_kill: Killing 1 processes (idle RFS by thread/sequence)

      Wed Oct 09 18:51:16 2013

      krsv_proc_kill: Killing 1 processes (idle RFS by thread/sequence)

      Wed Oct 09 18:51:16 2013

      RFS[68]: Assigned to RFS process 626724

      RFS[68]: Opened log for thread 1 sequence 4867 dbid 1547947009 branch 757523841

      RFS[69]: Assigned to RFS process 684156

      RFS[69]: Opened log for thread 1 sequence 4866 dbid 1547947009 branch 757523841

      RFS[70]: Assigned to RFS process 483332

      RFS[70]: Opened log for thread 1 sequence 4872 dbid 1547947009 branch 757523841

       

      ######### Primary database : archive log status #########

      FACILITY                 SEVERITY      MESSAGE_NUM ERROR_CODE CAL TO_CHAR(TIMESTAMP,'DD-MON-YYY MESSAGE

      ------------------------ ------------- ----------- ---------- --- ----------------------------- ------------------------------------------------------------

      Log Transport Services   Error               41532      16198 YES 09-OCT-2013 17:41:22          WARN: ARC1: Terminating pid 3600618 hung on an I/O operation

      Log Transport Services   Error               41533      16198 YES 09-OCT-2013 17:41:27          WARN: ARC1: Terminating pid 4071430 hung on an I/O operation

      Log Transport Services   Error               41536      16198 YES 09-OCT-2013 17:41:36          ARC1: Error 16198 due to hung I/O operation to LOG_ARCHIVE_D

                                                                                                                                           EST_2

      Log Transport Services   Error               41540      16198 YES 09-OCT-2013 17:41:36          WARN: ARC2: Terminating pid 3960900 hung on an I/O operation

      Log Transport Services   Error               41543      16198 YES 09-OCT-2013 17:41:41          WARN: ARC3: Terminating pid 3960900 hung on an I/O operation

      Log Transport Services   Error               41548      16198 YES 09-OCT-2013 17:46:37          WARN: ARC1: Terminating pid 3797106 hung on an I/O operation

      Log Transport Services   Error               41549      16198 YES 09-OCT-2013 17:46:42          WARN: ARC1: Terminating pid 3600622 hung on an I/O operation

      Log Transport Services   Error               41550      16198 YES 09-OCT-2013 17:46:46          WARN: ARC1: Terminating pid 4071432 hung on an I/O operation

      Log Transport Services   Error               41551      16198 YES 09-OCT-2013 17:46:51          WARN: ARC1: Terminating pid 4001810 hung on an I/O operation

        • 1. Re: Logs are not shipped to the DR database. Please help
          saurabh

          you should check the pinging status of the standby server to confirm the same. 

          • 2. Re: Logs are not shipped to the DR database. Please help
            mseberg

            Hello;

             

            I would check this:

             

             

             

             

            Best Regards

             

            mseberg

            • 3. Re: Logs are not shipped to the DR database. Please help
              Anar Godjaev

              HI,

               

              Solution oracle support:

               

              Apply Patch 11853815 or upgrade at least to 11.2.0.2.4 or 11.2.0.3.

               

              < MOS document content removed by moderator.   Posting such information to a publicly readable forum violates your Support contract privileges and could subject your entire organization to being banned from all Support.   Do NOT do that. >

               

              .

              .

               

              Message was edited by: rukbat

              • 4. Re: Logs are not shipped to the DR database. Please help
                BluShadow

                I have removed your response as you are not permitted to post the content of Oracle support documents on publicly accessible forums or the internet.

                 

                You may quote the bug/support number so the person may log on via their own support and find the information, but you must not post the content of such things.

                • 5. Re: Logs are not shipped to the DR database. Please help
                  Anar Godjaev

                  HI BluShadow

                   

                  Yes of course. Thank you for information.

                   

                  Best Regards

                  • 6. Re: Logs are not shipped to the DR database. Please help
                    Anar Godjaev

                    HI

                     

                    Apply Patch 11853815 or upgrade at least to 11.2.0.2.4 or 11.2.0.3.

                     

                    More information please oracle support Doc ID 1318372.1

                     

                    Thank you

                    • 7. Re: Logs are not shipped to the DR database. Please help
                      TzuWen, Lin

                      Hello :

                       

                      I have applied patch 11853815 . But The DG still can't works. Standby database can not receiving logs.

                       

                       

                      ############# MPLS VPN (Standby Database at IDC) #############

                      </u3/gaptmp>

                      </u3/gaptmp> scp -r PLMPRMY:/u2/src/p10098816_112020_AIX64-5L_7of7.zip .

                      oracle@plmprmy's password:

                      p10098816_112020_AIX64-5L_7of7.zip                                               23%   48MB 791.9KB/s   03:20 ETA

                      ############# MPLS VPN (Standby Database at IDC) #############

                       

                        

                       

                       

                      SQL> select to_char(sysdate ,'YYYY/MM/DD HH24:MI:SS') from dual;

                      select CLIENT_PROCESS,PROCESS,SEQUENCE#,STATUS,BLOCKS,BLOCK# from v$managed_standby ;

                      TO_CHAR(SYSDATE,'YY

                      -------------------

                      2013/10/10 19:00:21

                       

                       

                      SQL>

                       

                       

                      CLIENT_P PROCESS    SEQUENCE# STATUS           BLOCKS     BLOCK#

                      -------- --------- ---------- ------------ ---------- ----------

                      N/A      MRP0            4864 WAIT_FOR_GAP          0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      UNKNOWN  RFS             4881 IDLE                  0          0

                      UNKNOWN  RFS             4865 IDLE                  0          0

                      UNKNOWN  RFS             4882 IDLE                  0          0

                      LGWR     RFS             5020 IDLE                  1          1

                      UNKNOWN  RFS             5021 IDLE                  0          0

                      UNKNOWN  RFS             4880 IDLE                  0          0

                      UNKNOWN  RFS             4864 IDLE                  0          0

                      LGWR     RFS             5023 IDLE                  1          1

                      LGWR     RFS             5024 IDLE                  1          1

                       

                       

                      14 rows selected.

                       

                       

                       

                      SQL> select to_char(sysdate ,'YYYY/MM/DD HH24:MI:SS') from dual;

                      select CLIENT_PROCESS,PROCESS,SEQUENCE#,STATUS,BLOCKS,BLOCK# from v$managed_standby ;

                      TO_CHAR(SYSDATE,'YY

                      -------------------

                      2013/10/10 19:06:31

                       

                       

                      SQL>

                       

                       

                      CLIENT_P PROCESS    SEQUENCE# STATUS           BLOCKS     BLOCK#

                      -------- --------- ---------- ------------ ---------- ----------

                      N/A      MRP0            4864 WAIT_FOR_GAP          0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      ARCH     ARCH               0 CONNECTED             0          0

                      UNKNOWN  RFS             4881 IDLE                  0          0

                      UNKNOWN  RFS             4865 IDLE                  0          0

                      UNKNOWN  RFS             4882 IDLE                  0          0

                      LGWR     RFS             5020 IDLE                  1          1

                      LGWR     RFS             5025 IDLE                  1          1

                      UNKNOWN  RFS             4880 IDLE                  0          0

                       

                       

                      CLIENT_P PROCESS    SEQUENCE# STATUS           BLOCKS     BLOCK#

                      -------- --------- ---------- ------------ ---------- ----------

                      UNKNOWN  RFS             5021 IDLE                  0          0

                      UNKNOWN  RFS             4864 IDLE                  0          0

                      LGWR     RFS             5024 IDLE                  1          1

                       

                       

                      14 rows selected.

                       

                       

                       

                       

                      Standby can not receiving block (BLOCKS always "1").

                       

                      Please help.

                       

                       

                       

                      Tomorrow . I will install 11.2.0.3 in dbhome_2 , and prepare new instance to test dataguard.

                       

                      Thanks for help

                      • 8. Re: Logs are not shipped to the DR database. Please help
                        TzuWen, Lin

                        Logs are not shipped to the physical standby database [ID 1130523.1]

                         

                        ) Please work with your Network Administrator to make sure the following Firewall Features are disabled.

                         

                        • SQLNet fixup protocol 
                        • Deep Packet Inspection (DPI) 
                        • SQLNet packet inspection 
                        • SQL Fixup 
                        • SQL ALG (Juniper firewall)

                         

                        Disable SQL ALG .