1 2 Previous Next 15 Replies Latest reply on Jan 30, 2018 6:51 PM by Andris Perkons-Oracle

    Is there any relationship between Data Guard and RAC Heartbeat?

    rlarios

      good morning every body. into our company we have move from oracle 11g to oracle 12c. there is a 4 nodes cluster rac into the primary site, also into altern site we have same configuration. Into this RAC there are 4 databases. 2 of those databases were configured by using Oracle Active Data Guard.

      Everything seemed to be good, but one of those databases began to have low performance. Enterprise Manager Analysis mentioned the principal mistake was cluster latency.

      We looked for those problems, but we could not find anything.

      The cluster's waits grown up every day, users reclaimed to have low performance mainly on final process,

      which increased until 300% more process time before we migrated to Oracle 12c.

      One of the DBA's suggested the problem could be the Active Data Guard.

      So Active Data Guard was deactivated from those database.  suddenly the cluster's waits started to go down.

      The final process time went down. at the end operation, specially, final process recovered its performance.

       

      Now the question is... what is the relationship between Oracle Active Data Guard and Oracle RAC?

      I mean, I think Oracle dataguard use an isolate domaint in order to mantaind the health of cluster.

      Oracle Active Data Guard, in this case, is using a LAN in order to send data from primary site to remote site.

      How could OADG damage or hurt the performance into private cluster network?

        • 1. Re: Is there any relationship between Data Guard and RAC Heartbeat?
          jgarry

          I wouldn't know, but I could guess you had a pga problem.  https://www.scribd.com/presentation/115566234/Internals-of-Active-DataGuard

          Edit:  Slide 21 there seems to say, yes, there is some kind of relationship, since apply crash can shutdown other rac nodes.

          Also, which mode?  Maximum availability would imply any applying slow down would affect everything.

           

          Why did the DBA suggest it?

          • 2. Re: Is there any relationship between Data Guard and RAC Heartbeat?
            John Thorton

            rlarios wrote:

             

            good morning every body. into our company we have move from oracle 11g to oracle 12c. there is a 4 nodes cluster rac into the primary site, also into altern site we have same configuration. Into this RAC there are 4 databases. 2 of those databases were configured by using Oracle Active Data Guard.

            Everything seemed to be good, but one of those databases began to have low performance. Enterprise Manager Analysis mentioned the principal mistake was cluster latency.

            We looked for those problems, but we could not find anything.

            The cluster's waits grown up every day, users reclaimed to have low performance mainly on final process,

            which increased until 300% more process time before we migrated to Oracle 12c.

            One of the DBA's suggested the problem could be the Active Data Guard.

            So Active Data Guard was deactivated from those database. suddenly the cluster's waits started to go down.

            The final process time went down. at the end operation, specially, final process recovered its performance.

             

            Now the question is... what is the relationship between Oracle Active Data Guard and Oracle RAC?

            I mean, I think Oracle dataguard use an isolate domaint in order to mantaind the health of cluster.

            Oracle Active Data Guard, in this case, is using a LAN in order to send data from primary site to remote site.

            How could OADG damage or hurt the performance into private cluster network?

            Data Guard works by shipping archived REDO log files to remote site.

            Please explain how the action to copy archived REDO log file has any direct impact against OLTP activity.

            Is the disk subsystem adequate to support both OLTP & additional disk I/O required to copy archived REDO files?

            In other words is disk subsystem a performance bottleneck?

             

            during heaviest usage hours, what is the rate of REDO generation? How many Megabytes/hour of REDO is produced?

            • 3. Re: Is there any relationship between Data Guard and RAC Heartbeat?
              DavidMcWhinnie

              I would look at your network config and ensure your data guard setup is not using the same network as the RAC interconnect.  OEM is reporting cluster latency, so the network is the first place I would look.   Confirm jumbo frames and any other settings for RAC.  Also make sure the private network is really private and not used by/for anything else.  After that, I would look at data guard config.  Have you setup synchronous instead of async?

              • 4. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                rlarios

                In fact, The infraestructure is mounted on an ibm system called VIOS, cluster disk administrator is Veritas and the storage is a vmax 10K using disk arrays at 600GB connected to 8gb mds.

                • 5. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                  rlarios

                  At the begin, data guard was configured as MaxAvailabilty,

                  but, trying to identifiy the real cause of low performance,

                  the configuration was changed to MaxPerformance. At the end,

                  we decided to power data guard off.

                  • 6. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                    John Thorton

                    rlarios wrote:

                     

                    In fact, The infraestructure is mounted on an ibm system called VIOS, cluster disk administrator is Veritas and the storage is a vmax 10K using disk arrays at 600GB connected to 8gb mds.

                    Forgive me for being dense, but I fail to understand how above answers any of my 4 questions.

                     

                    During heaviest usage hours, what is the rate of REDO generation?

                     

                    How many Megabytes/hour of REDO is produced at peak load?

                    • 7. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                      SUPRIYO DEY

                      Is DG and Public network using the same NIC? First place to check that and second one is the How SAN is mapped to the Server.

                      • 8. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                        rlarios

                        According with network team and unix team, heart beat and  DG are using different NIC into the servers.

                        • 9. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                          rlarios

                          Data Guard works by shipping archived REDO log files to remote site.

                          Please explain how the action to copy archived REDO log file has any direct impact against OLTP activity.

                          A: --- This is my doubt. archived REDO is stored directly to disks Array. according to EM graphics, there were not problems on reads or writes to disk. But waits on cluster were higer than commit waits.

                          Is the disk subsystem adequate to support both OLTP & additional disk I/O required to copy archived REDO files?

                          A. ---- Yes. disks array is managed by veritas cluster file system.

                          In other words is disk subsystem a performance bottleneck?

                            A. --- No, as I told you before, reads and writes were not an issue

                          during heaviest usage hours, what is the rate of REDO generation? How many Megabytes/hour of REDO is produced?

                          A: ---- It is very strange, because this is a oltp database but, is not a too heavy oltp, redo logs files are defined 200MB the maximum number of redo files generated on one hour was 6 ....

                          • 10. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                            Andris Perkons-Oracle

                            So, if you can just "power off" Data Guard, do you actually need it?

                            No one is setting up Data Guard just for fun, usually this is done because there are some DR requirements that have to be met.

                             

                            Anyway, have a look at some Data Guard Best Practives Whitepapers. Both contain some pointers on parameter settings and tuning:

                             

                            http://www.oracle.com/technetwork/database/availability/sync-2437177.pdf

                            http://www.oracle.com/technetwork/database/availability/async-2587521.pdf

                             

                            Having a low network latency between both sites is pretty important. But one should also not underestimate the impact of slow disks on the standby site.

                            With 12c, there's a new mode (Fast Sync) that may improve performance for synchronous configurations.

                             

                            Andris

                            • 11. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                              rlarios

                              Hi Andris:

                               

                              it was a requeriment from company, as I told you before, we have another database using Active Data Guard, and it has no problems. We turned off data guard, but now we will change replication methode using Golden Gate while we solve this issue.

                              We will check those document you mentioned

                              • 12. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                                DavidMcWhinnie

                                rlarios wrote:

                                 

                                According with network team and unix team, heart beat and DG are using different NIC into the servers.

                                I wouldn't take the network and unix team at their word.  I would logon to the machine and confirm it.  Also just because they configured these separate networks, it doesn't mean the database is configured to use them.  Post results of your network config and the Oracle settings that show those networks are being used.

                                • 13. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                                  Hemant K Chitale

                                  Active Data Guard allows queries on the Standby database.

                                   

                                  Is your cluster latency on the Primary database ?  That has no relationship with ADG on the Standby.

                                   

                                  As has been mentioned, check which networks are being used -- the cluster interconnect must be a private network not shared with any other traffic.

                                   

                                   

                                  Hemant K Chitale

                                  • 14. Re: Is there any relationship between Data Guard and RAC Heartbeat?
                                    BPeaslandDBA

                                    There is a phrase that comes to my mind... "correlation does not imply causation".

                                     

                                    Active Data Guard has nothing to do with your primary database. ADG, the part that let's you open a physical standby database while still applying redo from the primary, runs entirely on the standby database.

                                     

                                    What does happen is that the primary ships redo to the standby. This involves two processes. RFS on the standby and either ARCn or NSSn/NSAn on the primary. See this for diagrams of how redo is shipped to the standby. Standby Redo Logs – Why and How?

                                     

                                    Nothing involved in this has anything to do with:

                                     

                                    Enterprise Manager Analysis mentioned the principal mistake was cluster latency.

                                     

                                    The only possible way for them to collide is if, as others have said, the private network is not truly private and is somehow shared with the transport of redo between systems. I agree with DavidMcWhinnie..don't take their word for it. Verify yourself. Once you have confirmed that the private network is not being shared, then the only conclusion is that DG had nothing to do with the problem.

                                     

                                    Did you drill down into that cluster latency? How big was the latency? What was the latency on? Was it for specific types of data blocks? Were all block transfers affected similarly? Was it just one or two nodes in the cluster or were all the cluster nodes similarly affected? If you're using EM for analysis like this, then you must be licensed for the Diag Pack. Which means you're also licensed for running AWR reports. The cluster AWR reports provide a ton of information to drill down into this type of problem.

                                     

                                    Cheers,
                                    Brian

                                    1 2 Previous Next