6 Replies Latest reply: Mar 25, 2013 9:29 PM by Hemant K Chitale RSS

    ARCH wait on SENDREQ

    Moazzam
      We are using Oracle 10g on Linux:

      The R&D shows that the main reason for ARCH wait on SENDREQ wait is slow netwrok.
      1-Can I get the performance of network (bytes sent per second) from the AWR report?
      2-Since the wait is related to archiving, do you agree that the Log_archive_dest_1 is the only culprit causing this wait event and Log_archive_dest_2 and Log_archive_dest_3 has nothing to do with this wait.
      3-The redo size is 893,323.09 per second. What should be the redo per second for a good system. Is there any way to reduce it.


      Followings are the Top 5 Timed Events in AWR report:
      Event                           Waits           Time(s)          Avg     Wait(ms)      % Total Call Time      Wait Class 
      ARCH wait on SENDREQ      43,206           10,079           233                        27.5                     Network 
      log file sync                        356,389              9,505           27                     25.9                     Commit 
      direct path read                1,719,611              6,647           4                     18.1                     User I/O 
      db file sequential read        697,323              5,602           8                     15.3                     User I/O 
      CPU time                                        5,355                                    14.6   
      log_archive_dest_1 LOCATION=+g3_recovery_area    
      log_archive_dest_2 SERVICE=abc VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) LGWR ASYNC DB_UNIQUE_NAME=cc    
      log_archive_dest_3 SERVICE=def VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) LGWR ASYNC DB_UNIQUE_NAME=PC 
        • 1. Re: ARCH wait on SENDREQ
          Salman Qureshi
          Hi,
          The R&D shows that the main reason for ARCH wait on SENDREQ wait is slow netwrok.
          Then tune your network first of all.
          1-Can I get the performance of network (bytes sent per second) from the AWR report?
          Some network related stats are there, but not enough to guage your network performance. you can install OSWatcher which is quite handy for DBAs. It monitors a lot of things including network.
          2-Since the wait is related to archiving, do you agree that the Log_archive_dest_1 is the only culprit causing this wait event and Log_archive_dest_2 and Log_archive_dest_3 has >nothing to do with this wait.
          All 3 destinations are considered for this event. If network is culprit according to your investigaiton then log_atchive_dest_1 may not be the culprit.
          3-The redo size is 893,323.09 per second. What should be the redo per second for a good system. Is there any way to reduce it.
          Redo means the data change made to your data so there is no guage for a "good" system measurement for this. More changes to data (insert/update/delete), more redo, less changes, less redo.
          Followings are the Top 5 Timed Events in AWR report:
          How much time does this AWR report cover?

          Salman

          Edited by: Salman Qureshi on Mar 25, 2013 4:09 PM
          • 2. Re: ARCH wait on SENDREQ
            Moazzam
            >
            How much time does this AWR report cover?
            >

            One Hour.

            >
            All 3 destinations are considered for this event.
            >

            Log_archive_dest_2 and Log_archive_dest_3 are using Asynch log writer process to asynchronously transmit the redo to remote destinations. Since they are asynchronous, so Log writer shall not be waiting to transmit redo. These destinations should have caused the LNS wait on SENDREQ event instead of ARCH wait on SENDREQ.

            Further analysis shows that we have higher number of commits. Do you think, these are also contributing to LNS wait on SENDREQ wait in addition to the log file sync wait event.
            user calls per second = 990.34
            user commits per second = 102.94
            If we reduce the size of redo log file, can it help ups to reduce the ARCH wait on SENDREQ waits.
            • 3. Re: ARCH wait on SENDREQ
              Salman Qureshi
              Hi,
              See MOS note ID 233491.1 for details of data guard waits and this will give you a good insight.
              Since they are asynchronous, so Log writer shall not be waiting to transmit redo. These destinations should have caused the LNS wait on SENDREQ event instead of ARCH wait >on SENDREQ.
              Writing asynchronously means that it will not wait for first transfer to finish to send next data over the network (and will not block the transfer), but it will still keep an eye finishing of the task and hence would know how fast data was transferred and how much wait was done. See the MOS note ID 241925.1 which is for 9i, but probably suites here too. >
              Further analysis shows that we have higher number of commits. Do you think, these are also contributing to LNS wait on SENDREQ wait in addition to the log file sync wait event.
              This note tells that this even occurs during log file switch. Log file sync is certainly caused by heave commits, but ARCH wait on SENDREQ may not.

              If we reduce the size of redo log file, can it help ups to reduce the ARCH wait on SENDREQ waits.
              Probably not. You are saying yourself that problem is at network side then the redo log size does not look relevant to this.

              Salman
              • 4. Re: ARCH wait on SENDREQ
                Moazzam
                >
                Writing asynchronously means that it will not wait for first transfer to finish to send next data over the network (and will not block the transfer), but it will still keep an eye finishing of the task and hence would know how fast data was transferred and how much wait was done
                >

                Do not you think that such waits should be added in the LGWR wait on SENDREQ event instead of ARCH wait on SENDREQ given the fact that we are using Log writer process to transmit redo to stand by database instead of archival process.

                >
                Probably not. You are saying yourself that problem is at network side then the redo log size does not look relevant to this.
                >

                Our network administrator is saying that network speed is fine. There is no issue with it. My below statement meant that I had gone through some articles about this error, and they said that this issue was related to slow network (poor communication):
                The R&D shows that the main reason for ARCH wait on SENDREQ wait is slow netwrok. 
                • 5. Re: ARCH wait on SENDREQ
                  Salman Qureshi
                  Hi,
                  Do not you think that such waits should be added in the LGWR wait on SENDREQ event instead of ARCH wait on SENDREQ given the fact that we are using Log writer process to >transmit redo to stand by database instead of archival process.
                  Makes sense I think, but just to isolate the problem, ss it possible if you can disable remote destinations by setting log_Archive_dest_state_2 and log_Archive_dest_state_3 for one hour (during AWR snap interval). And enable it after 1 hour and check AWR report if it still shows same wait event at the top?

                  Salman
                  • 6. Re: ARCH wait on SENDREQ
                    Hemant K Chitale
                    The redo size is 893,323.09 per second
                    That works out to more than 3GB/hour.
                    That would require a Network Bandwidth of 10-12Mbps on average. During peak periods, the requirement could be much higher. You need to size the network bandwidth to cover a reasonable ratio of the peak requirements (e.g. 90th percentile if you measure redo generation rates every 10minutes over a few days).
                    For example, a database may have a peak redo generation of 2GB/minute but a mean (average) rate of 50MB/minute !! When the peaks occur, redo transmission suffers


                    Hemant K Chitale