6 Replies Latest reply: Mar 17, 2013 12:00 PM by jhall RSS

    ORA-16724/ORA-16783: cannot resolve gap

    jhall
      Primary Database: 2 node rac cluster, 11.2.0.2 GI and RDBMS
      Standby Database: single instance 11.2.0.2
      OS: AIX 7.1

      Hello, I'm in the middle of setting up a data guard system, and am seeing a 16783 and a 16724. The system is saying it cannot resolve a gap. I performed this query to find the archive sequences that were needed: select * from v$archive_gap. 2 archive sequences were missing. I restored both of them to the primary system. 1 of the sequences was automatically recovered by the data guard system, the other was not. I manually copied the archive log that was not automatically recovered to the standby system. When I perform the select * from v$archive_gap now, no rows are returned. All archive log files that are currently in the primary system's archive directory have been copied to the standby system.

      Please let me know if anyone has an idea where the problem could be.

      Thanks
        • 1. Re: ORA-16724/ORA-16783: cannot resolve gap
          damorgan
          It would be very helpful if we could see the actual error messages rather than your interpretation of them.

          Personally I'd burn it to the ground and do it all again properly. Following the written docs for a Data Guard physical standby you would not have had a gap requiring manual intervention.
          • 2. Re: ORA-16724/ORA-16783: cannot resolve gap
            mseberg
            Hello;

            Those are tough errors. I have yet to see anybody fix them, repair the standby so its good. In a nutshell the base for the Standby is bad. At the point the Standby would start applying the logs it cannot because it is missing Archive from before that point.

            You can spend many hours trying to add those Archive logs so you apply will work. Most likely you will crew up a lot of time for nothing. Kind of like throwing good money after bad.

            Your best option is to cut your loses and redo the Standby. Nobody in your shoes wants to hear that, but there it is.

            Best Regards

            mseberg
            • 3. Re: ORA-16724/ORA-16783: cannot resolve gap
              jhall
              damorgan wrote:
              It would be very helpful if we could see the actual error messages rather than your interpretation of them.
              The actual error messages are in the subject line of my post.
              damorgan wrote:
              Following the written docs for a Data Guard physical standby you would not have had a gap requiring manual intervention.
              Link to the Oracle whitepaper I used for this installation: http://www.oracle.com/us/solutions/sap/wp-ora4sap-dataguard11g-303811.pdf

              I encountered this error because the whitepaper specifies to run a cold backup and copy it to the standby site. The source system is a multiple TB production system. A long outage for a cold backup/system copy is not an option. I had to use hot incremental backups to get the copy as close to the source as possible, and let DG sync the delta.
              • 4. Re: ORA-16724/ORA-16783: cannot resolve gap
                jhall
                Hi mseberg,

                To your point:
                mseberg wrote:
                Your best option is to cut your loses and redo the Standby. Nobody in your shoes wants to hear that, but there it is.
                Unfortunately, this was a production system and I had a limited amount of time to initiate the data guard system. The system copy process takes a long time because the source is so large, and I wouldn't have been able to fit it into my window.

                As I only had one option for getting this system up in time, I did try to resolve this gap manually. I found something very interesting during this process. I ran this query on the standby site multiple times: 'select * from v$archive_gap;', and it returned no rows. But, the archive logs kept piling up and were not being applied. I eventually found in the standby DB alert log that the database was waiting for a previous archive sequence that did not exist in my archive mount point. V$archive_gap did not show that this sequence was required, but the alert log said otherwise. On the primary, I pulled this sequence up from a tape backup, and it applied to the standby and resolved the gap. I guess the moral of the story for anyone else who encounters the ORA-16724/ORA-16783 errors... v$archive_gap may not show a complete picture of all missing archive logs. Check the alert log and verify that the archive sequence required by the standby really exists in your archive mount point.

                Mseberg, thanks also for your reference to using incremental backups to "catch up" the standby db with the primary db in a previous post of mine. I used this concept in preparing my standby copy. I took a hot lev 0 backup that ran through the night, and then took several hot lev 1 backups throughout the day prior to the conversion window. When it came time to start the system copy, I used the 'dorecover' option in my rman duplicate command (duplicate target database for standby nofilenamecheck dorecover;). Your idea prompted me to use the incremental backups, and this ended up saving a lot of time during this process.
                • 5. Re: ORA-16724/ORA-16783: cannot resolve gap
                  mseberg
                  Hello again;

                  Active duplication might be an option, The main concern is load will be placed on the network and source host during this process.

                  In theory you could set the "RATE Channel Parameter" in RMAN to control this :

                  http://docs.oracle.com/cd/E11882_01/backup.112/e10642/rcmtunin.htm#BABDCEHG

                  Best Regards

                  mseberg
                  • 6. Re: ORA-16724/ORA-16783: cannot resolve gap
                    jhall
                    Nice, I didn't realize you could throttle the channel rate, that could be very useful. We've also seen situations in the past where a full lev 0 backup causes a performance degradation while the system is under heavy load. This parameter could help us to minimize the impact in this situation. Thanks for the tip.