11 Replies Latest reply on Mar 8, 2013 7:22 AM by thodi

    RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately

    thodi
      Hi,

      I may be missing something obvious here. I am taking RMAN backups using Veritas NetBackup, the NetBackup policy uses "inifinity" retention period, RMAN is configured to use a recovery windows fo 31 days (this is a NetBackup best practice for managing expiry). Backups run fine and after they're done, I see lots of reclaimable space in my FRA, but as soon as I run "crosscheck backup", every backup I have ever done is marked EXPIRED.

      RMAN settings:
      RMAN> show all;
      
      starting full resync of recovery catalog
      full resync complete
      RMAN configuration parameters for database with db_unique_name BOPA are:
      CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 31 DAYS;
      CONFIGURE BACKUP OPTIMIZATION ON;
      CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
      CONFIGURE CONTROLFILE AUTOBACKUP OFF;
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
      CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 3 BACKUP TYPE TO BACKUPSET;
      CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE MAXSETSIZE TO UNLIMITED; # default
      CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
      CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
      CONFIGURE COMPRESSION ALGORITHM 'BZIP2'; # default
      CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
      CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.1.0/db_1/dbs/snapcf_bopa2.f'; # default
      Crosscheck:
      RMAN> crosscheck backup;
      [...]
      crosschecked backup piece: found to be 'EXPIRED'
      backup piece handle=al_2298_1_809254750 RECID=2269 STAMP=809254789
      Crosschecked 19 objects
      The backup with handle al_2298_1_809254750 was taken last night, for example, with the RMAN settings listed above.

      Any ideas? My FRA has grown to 1.5TB, I have disabled automatic crosscheck for now -- but I will need to reenable it.

      Oracle 11.1.0.7 SE, 2 node Linux RAC, NetBackup 7.5
        • 1. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
          mseberg
          Hello;

          Its looks like your "DEFAULT DEVICE TYPE" is Disk. RMAN will mark "Expired" any backup no longer exists on disk, but is still listed in the repository.

          CROSSCHECK command updates the repository status for a backup to EXPIRED when it cannot locate it at the location to which it was backed up.

          So if a backup was deleted from disk that you have on tape this would occur.

          Don't overuse CROSSCHECK. CROSSCHECK compares the RMAN catalog entries. This is useful when RMAN thinks a file is missing. If you need confirm recovery catalog is synchronized with corresponding data on disk or in the media then use it. Otherwise if you have no mismatched objects all you do is change the status of files to "EXPIRED".

          That said you still want to run CROSSCHECK frequently enough so that the LIST EXPIRED BACKUP command is still useful. If you manually delete any kind of RMAN related backup file at the OS level, then you will need to use CROSSCHECK at some point.

          If you use a backup retention policy and an archive log deletion policy with FRA maintenance becomes almost automatic and then there's no need to constantly execute CROSSCHECK to verify your backups.

          Best Regards

          mseberg
          1 person found this helpful
          • 2. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
            Stefan Abraham
            Hi

            Seems you have a confusion between "obsolete" and "Expired"
            check with report obsolete.
            It will show the backups available according to your retention policy.

            Thanks
            Stefan
            1 person found this helpful
            • 3. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
              Hemant K Chitale
              Backups run fine and after they're done, I see lots of reclaimable space in my FRA
              Are the backups going to FRA (on disk) or tape directly ? Does the reclaimable space in the FRA change when you run a backup ?


              Are you running the CROSSCHECK command when connected to NetBackup (with the NetBackup Environment parameters configured) ? Else it would not find backuppieces written to NetBackup if it is not connected to NetBackup.


              Hemant K Chitale
              1 person found this helpful
              • 4. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                thodi
                FYI: I won't be back at the office until tomorrow. Thank you for all the insight so far, I'll get back to you.
                • 5. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                  thodi
                  Hemant K Chitale wrote:
                  Backups run fine and after they're done, I see lots of reclaimable space in my FRA
                  Are the backups going to FRA (on disk) or tape directly ? Does the reclaimable space in the FRA change when you run a backup ?
                  The backups are going directly to tape. After running a backup, I see reclaimable space à la:
                  FILE_TYPE                        PERCENT_SPACE_USED    GB_USED GB_RECLAIMABLE NUMBER_OF_FILES
                  -------------------------------- ------------------ ---------- -------------- ---------------
                  ARCHIVED LOG                                  35.83      716.6          709.6           10390
                  BACKUP PIECE                                  22.85        457              0              70
                  IMAGE COPY                                    18.42      368.4            214              16
                  TOTAL                                          77.1       1542          923.6           10476
                  But after running the crosscheck, all reclaimable columns are back to zero.
                  Are you running the CROSSCHECK command when connected to NetBackup (with the NetBackup Environment parameters configured) ? Else it would not find backuppieces written to NetBackup if it is not connected to NetBackup.
                  I ran 'allocate channel for maintenance type sbt_tape;' before doing the crosscheck.
                  • 6. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                    thodi
                    Stefan Abraham wrote:
                    Seems you have a confusion between "obsolete" and "Expired"
                    check with report obsolete.
                    It will show the backups available according to your retention policy.
                    report obsolete returns that no backups are obsolete at the moment, you are right. I guess I will have to wait what happens after 31 days :-)
                    • 7. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                      mseberg
                      Hello Again;

                      This might be worth a look too.

                      'Crosscheck Backup' incorrectly marks 'EXPIRED' eventhough backup files are physically present in TAPE drive [ID 1124254.1]

                      Best Regards

                      mseberg
                      1 person found this helpful
                      • 8. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                        thodi
                        mseberg wrote:
                        Hello;

                        Its looks like your "DEFAULT DEVICE TYPE" is Disk. RMAN will mark "Expired" any backup no longer exists on disk, but is still listed in the repository.

                        CROSSCHECK command updates the repository status for a backup to EXPIRED when it cannot locate it at the location to which it was backed up.
                        I ran 'allocate channel for maintenance type sbt_tape;' before the crosscheck, I was under the impression that this would connect me to the NetBackup database. I forgot to mention that in my initial post.

                        Don't overuse CROSSCHECK. CROSSCHECK compares the RMAN catalog entries. This is useful when RMAN thinks a file is missing. If you need confirm recovery catalog is synchronized with corresponding data on disk or in the media then use it. Otherwise if you have no mismatched objects all you do is change the status of files to "EXPIRED".

                        That said you still want to run CROSSCHECK frequently enough so that the LIST EXPIRED BACKUP command is still useful. If you manually delete any kind of RMAN related backup file at the OS level, then you will need to use CROSSCHECK at some point.

                        If you use a backup retention policy and an archive log deletion policy with FRA maintenance becomes almost automatic and then there's no need to constantly execute CROSSCHECK to verify your backups.
                        I think I may have had a misconception about corsscheck. I thought it was the way to actually expire backup images. I will have to reread a lot, I guess. The funny thing is this scheme worked flawlessly in another environment.
                        • 9. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                          thodi
                          mseberg wrote:
                          This might be worth a look too.

                          'Crosscheck Backup' incorrectly marks 'EXPIRED' eventhough backup files are physically present in TAPE drive [ID 1124254.1]
                          Thanks, I will add a "show channel" command to my backup script and see what it gives. Running it in rman by hand gives
                          RMAN> show channel;
                          
                          RMAN configuration parameters for database with db_unique_name BOPA are:
                          RMAN configuration has no stored or default parameters
                          
                          RMAN> 
                          • 10. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                            Hemant K Chitale
                            I ran 'allocate channel for maintenance type sbt_tape;' before doing the crosscheck.
                            So, in theory, you are running the CROSSCHECK against Tape.
                            But after running the crosscheck, all reclaimable columns are back to zero
                            However, the Reclaimable Space on Disk seems to be getting cleared.
                            After running a backup, I see reclaimable space
                            I presume that you do not have a DELETE INPUT in your BACKUP ARCHIVELOG command ?


                            I think you shouldn't be running CROSSCHECK against the tape.
                            You might run CROSSCHECK against disk.


                            Hemant K Chitale
                            • 11. Re: RMAN + Veritas NetBackup: Backups are marked as EXPIRED immediately
                              thodi
                              Hemant K Chitale wrote:
                              I ran 'allocate channel for maintenance type sbt_tape;' before doing the crosscheck.
                              So, in theory, you are running the CROSSCHECK against Tape.
                              That's what I thought as well.
                              But after running the crosscheck, all reclaimable columns are back to zero
                              However, the Reclaimable Space on Disk seems to be getting cleared.
                              No, the space remains allocated, it's just not reclaimable:

                              Before cosscheck:
                              FILE_TYPE                        PERCENT_SPACE_USED    GB_USED GB_RECLAIMABLE NUMBER_OF_FILES
                              -------------------------------- ------------------ ---------- -------------- ---------------
                              ARCHIVED LOG                                  44.83     672.45         667.35            9810
                              BACKUP PIECE                                  27.96      419.4          418.2              58
                              IMAGE COPY                                    24.54      368.1          368.1              16
                              TOTAL                                         97.33    1459.95        1453.65            9884
                              After crosscheck:
                              FILE_TYPE                        PERCENT_SPACE_USED    GB_USED GB_RECLAIMABLE NUMBER_OF_FILES
                              -------------------------------- ------------------ ---------- -------------- ---------------
                              ARCHIVED LOG                                  44.91     673.65              0            9830
                              BACKUP PIECE                                  27.96      419.4              0              58
                              IMAGE COPY                                    24.56      368.4              0              16
                              TOTAL                                         97.43    1461.45              0            9904
                              After running a backup, I see reclaimable space
                              I presume that you do not have a DELETE INPUT in your BACKUP ARCHIVELOG command ?
                              Nope. I want as little interference with FRA contents as possible :-)
                              I think you shouldn't be running CROSSCHECK against the tape.
                              You might run CROSSCHECK against disk.
                              I shall refrain from running crosscheck and just see what "report obsolete" tells me after 31 days of backups.