9 Replies Latest reply: Mar 22, 2013 12:44 AM by TSharma-Oracle RSS

    Block corruption error in system datafile

    996245
      Hi Guru,
      There is a problem in system datafile of production database, one of the block has been corrupted.we have set the configuration of RMAN with SBT_TAPE for backup. when we try to recover using then following commands then we get some error. Please help to solve the issue.database 1s 10g on linux environment.

      RMAN> RUN
      {
      ALLOCATE CHANNEL c1 DEVICE TYPE disk;
      blockrecover datafile 3 block 39830;
      }2> 3> 4> 5>

      allocated channel: c1
      channel c1: sid=1417 devtype=DISK

      Starting blockrecover at 21-MAR-13

      released channel: c1
      RMAN-00571: ===========================================================
      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
      RMAN-00571: ===========================================================
      RMAN-03002: failure of blockrecover command at 03/21/2013 13:02:10
      RMAN-06026: some targets not found - aborting restore
      RMAN-06100: no channel to restore a backup or copy of datafile 3

      RMAN>


      **********************************************

      RMAN> blockrecover datafile 3 block 39830;

      Starting blockrecover at 21-MAR-13
      allocated channel: ORA_DISK_1
      channel ORA_DISK_1: sid=1417 devtype=DISK
      allocated channel: ORA_DISK_2
      channel ORA_DISK_2: sid=2351 devtype=DISK
      allocated channel: ORA_DISK_3
      channel ORA_DISK_3: sid=1452 devtype=DISK
      allocated channel: ORA_DISK_4
      channel ORA_DISK_4: sid=1270 devtype=DISK
      allocated channel: ORA_DISK_5
      channel ORA_DISK_5: sid=1865 devtype=DISK
      allocated channel: ORA_DISK_6
      channel ORA_DISK_6: sid=2064 devtype=DISK
      allocated channel: ORA_DISK_7
      channel ORA_DISK_7: sid=1734 devtype=DISK
      allocated channel: ORA_DISK_8
      channel ORA_DISK_8: sid=1574 devtype=DISK
      released channel: ORA_DISK_1
      released channel: ORA_DISK_2
      released channel: ORA_DISK_3
      released channel: ORA_DISK_4
      released channel: ORA_DISK_5
      released channel: ORA_DISK_6
      released channel: ORA_DISK_7
      released channel: ORA_DISK_8
      RMAN-00571: ===========================================================
      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
      RMAN-00571: ===========================================================
      RMAN-03002: failure of blockrecover command at 03/21/2013 13:19:59
      ORA-19554: error allocating device, device type: SBT_TAPE, device name:
      ORA-27211: Failed to load Media Management Library
      Additional information: 2

      RMAN>

      RMAN> show all;

      RMAN configuration parameters are:
      CONFIGURE RETENTION POLICY TO REDUNDANCY 7;
      CONFIGURE BACKUP OPTIMIZATION OFF;
      CONFIGURE DEFAULT DEVICE TYPE TO DISK;
      CONFIGURE CONTROLFILE AUTOBACKUP ON;
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/d02/rman/ctl_%F';
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'SBT_TAPE' TO '%F';
      CONFIGURE DEVICE TYPE DISK PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
      CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
      CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/d02/rman/%T_%d_%s_%p';
      CONFIGURE MAXSETSIZE TO UNLIMITED;
      CONFIGURE ENCRYPTION FOR DATABASE OFF;
      CONFIGURE ENCRYPTION ALGORITHM 'AES128';
      CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
      CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/d02/rman/snapcf_PROD.f';
        • 1. Re: Block corruption error in system datafile
          Mihael
          use "list backup" command to find on what device you have required backup, if it is tape device, you should allocate tape device, not disk
          • 2. Re: Block corruption error in system datafile
            996245
            I tried the same but still the same error.No luck.
            • 3. Re: Block corruption error in system datafile
              TSharma-Oracle
              How are you allocating TAPE? Post a full command you are using to recover.

              Also, Post your backup script you use daily to take a backup.

              Edited by: TSharma on Mar 22, 2013 12:52 AM
              • 4. Re: Block corruption error in system datafile
                996245
                Please find the below backup and recovery scripts.

                Daily backup script
                ====================
                run {
                allocate channel 'dev_0' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_1' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_2' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_3' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_4' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_5' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_6' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                allocate channel 'dev_7' type 'sbt_tape'
                parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                delete noprompt archivelog until time 'sysdate-2';
                crosscheck archivelog all;
                sql 'alter system archive log current';
                backup as compressed backupset
                format 'Monday-ERP-RMAN-Daily<PROD_%s:%t:%p>.dbf'
                database;
                backup as compressed backupset
                format 'Monday-ERP-RMAN-Daily<PROD_%s:%t:%p>.dbf'
                archivelog all;
                backup
                format 'Monday-ERP-RMAN-Daily<PROD_%s:%t:%p>.dbf'
                current controlfile;
                }


                we tried the following command as well.

                RMAN> RUN
                {
                ALLOCATE CHANNEL c1 DEVICE 'sbt_tape';
                blockrecover datafile 3 block 39830;
                }

                but the same error as below.

                RMAN> show all;

                RMAN configuration parameters are:
                CONFIGURE RETENTION POLICY TO REDUNDANCY 7;
                CONFIGURE BACKUP OPTIMIZATION OFF;
                CONFIGURE DEFAULT DEVICE TYPE TO DISK;
                CONFIGURE CONTROLFILE AUTOBACKUP ON;
                CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/d02/rman/ctl_%F';
                CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'SBT_TAPE' TO '%F';
                CONFIGURE DEVICE TYPE DISK PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
                CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
                CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
                CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
                CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
                CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE 'SBT_TAPE' TO 1;
                CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/d02/rman/%T_%d_%s_%p';
                CONFIGURE MAXSETSIZE TO UNLIMITED;
                CONFIGURE ENCRYPTION FOR DATABASE OFF;
                CONFIGURE ENCRYPTION ALGORITHM 'AES128';
                CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
                CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/d02/rman/snapcf_PROD.f';

                RMAN>




                also tried the following command
                =======================

                RMAN> RUN
                {
                ALLOCATE CHANNEL c1 DEVICE TYPE disk;
                blockrecover datafile 3 block 39830;
                }2> 3> 4> 5>

                allocated channel: c1
                channel c1: sid=1417 devtype=DISK

                Starting blockrecover at 21-MAR-13

                released channel: c1
                RMAN-00571: ===========================================================
                RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                RMAN-00571: ===========================================================
                RMAN-03002: failure of blockrecover command at 03/21/2013 13:02:10
                RMAN-06026: some targets not found - aborting restore
                RMAN-06100: no channel to restore a backup or copy of datafile 3

                RMAN>


                **********************************************

                RMAN> blockrecover datafile 3 block 39830;

                Starting blockrecover at 21-MAR-13
                allocated channel: ORA_DISK_1
                channel ORA_DISK_1: sid=1417 devtype=DISK
                allocated channel: ORA_DISK_2
                channel ORA_DISK_2: sid=2351 devtype=DISK
                allocated channel: ORA_DISK_3
                channel ORA_DISK_3: sid=1452 devtype=DISK
                allocated channel: ORA_DISK_4
                channel ORA_DISK_4: sid=1270 devtype=DISK
                allocated channel: ORA_DISK_5
                channel ORA_DISK_5: sid=1865 devtype=DISK
                allocated channel: ORA_DISK_6
                channel ORA_DISK_6: sid=2064 devtype=DISK
                allocated channel: ORA_DISK_7
                channel ORA_DISK_7: sid=1734 devtype=DISK
                allocated channel: ORA_DISK_8
                channel ORA_DISK_8: sid=1574 devtype=DISK
                released channel: ORA_DISK_1
                released channel: ORA_DISK_2
                released channel: ORA_DISK_3
                released channel: ORA_DISK_4
                released channel: ORA_DISK_5
                released channel: ORA_DISK_6
                released channel: ORA_DISK_7
                released channel: ORA_DISK_8
                RMAN-00571: ===========================================================
                RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                RMAN-00571: ===========================================================
                RMAN-03002: failure of blockrecover command at 03/21/2013 13:19:59
                ORA-19554: error allocating device, device type: SBT_TAPE, device name:
                ORA-27211: Failed to load Media Management Library
                Additional information: 2

                RMAN>
                • 5. Re: Block corruption error in system datafile
                  TSharma-Oracle
                  Run this below statement , you should get what you are looking for.

                  run {
                  allocate channel 'dev_0' type 'sbt_tape'
                  parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                  blockrecover datafile 3 block 39830;
                  }
                  • 6. Re: Block corruption error in system datafile
                    996245
                    Sorry I forgot to mention that I have already executed the following command.

                    run {
                    allocate channel 'dev_0' type 'sbt_tape'
                    parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                    blockrecover datafile 3 block 39830;
                    }

                    but same error.
                    • 7. Re: Block corruption error in system datafile
                      606331
                      >
                      RMAN> RUN
                      {
                      ALLOCATE CHANNEL c1 DEVICE TYPE disk;
                      blockrecover datafile 3 block 39830;
                      }2> 3> 4> 5>
                      if your using netbackup..above command won't work.. i hope.....

                      which type of net backup your using....?

                      if your using veritas... then fallow below steps...

                      $cat /usr/openv/netbackup/bp.conf

                      you may find master server details...otherwise get the details from your netbackup team....

                      then try below steps ..it may helpful to you....

                      step 1:

                      run {
                      allocate channel c1 type 'sbt_tape';
                      send 'NB_ORA_SERV=your_bkp_master, NB_ORA_CLIENT=your_server';
                      list backup;
                      release channel c1;
                      }

                      you may find your sucessfull backup tags over there....

                      step 2:
                      try to recover by using sucessful backup tag....

                      run {
                      allocate channel c1 type 'sbt_tape';
                      send 'NB_ORA_SERV=your_bkp_master, NB_ORA_CLIENT=your_server';
                      BLOCKRECOVER CORRUPTION LIST FROM TAG "xx";
                      release channel c1;
                      }

                      then check .... sql>select * from v$backup_corruption;

                      hope it may helpful to you....
                      • 8. Re: Block corruption error in system datafile
                        TSharma-Oracle
                        Run this: with crosscheck backup clause. and Post the output


                        run {
                        allocate channel 'dev_0' type 'sbt_tape'
                        parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Monday-ERP-RMAN-Daily)';
                        crosscheck backup;
                        blockrecover datafile 3 block 39830;
                        }
                        • 9. Re: Block corruption error in system datafile
                          996245
                          The net backup is "HP data protector". Actually the issue was related to Media Library. We also raised SR for the same issue. Oracle recommend the following scripts.
                          Problem was not in oracle it was in Media Library now resolved..

                          Thanks for your expert support.


                          ********** Restore & Recover database (Datafile System03.dbf) from HP DP for block recover *****************************
                          run {
                          allocate channel 'dev_0' type 'sbt_tape'
                          parms *'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so*,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_1' type 'sbt_tape'
                          parms *'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so*,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_2' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_3' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_4' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_5' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_6' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          allocate channel 'dev_7' type 'sbt_tape'
                          parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=PROD,OB2BARLIST=Wednesday-ERP-RMAN-Daily)';
                          blockrecover datafile 3 block 39830;
                          }


                          Thanks to everyone.