3 Replies Latest reply: Jan 31, 2013 8:40 AM by 652072 RSS

    RMAN backup failure

    GeetaM
      Hi Guru

      RMAN backup is failing daily, i have googled it but still didnt find helpful, could you please assist me, below is the log file of RMAN -
      + /bin/sh -x bin/rman_dbf_bck.sh ***** ARCH INCR RAC_NO 
      PARAMETER=
      + [ -z ***** ] 
      PARAMETER= ***** 
      + shift 
      + [ -z ARCH ] 
      PARAMETER= *****  ARCH 
      + shift 
      + [ -z INCR ] 
      PARAMETER= *****  ARCH  INCR 
      + shift 
      + [ -z RAC_NO ] 
      PARAMETER= *****  ARCH  INCR  RAC_NO 
      + shift 
      + [ -z  ] 
      + sudo -u oracle /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh ***** ARCH INCR RAC_NO 
      
      Tue Jan 29 19:00:13 EST 2013 ----------------Beginning of Script------------
      Script name: rman_dbf_bck.sh
      
      ORACLE_SID:  *****
      HOST NAME:   dukeinfs01
      ORACLE_HOME: /opt/u01/app/oracle/product/11.2.0.3/DB
      DB_MODE:     ARCH
      RMAN_LV:     INCR
      RAC_YORN:    RAC_NO
      
      /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh: [[: not found
      /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh: [[: not found
      /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh: [[: not found
      /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh: [[: not found
      COLD database backup requested for *****
      
      %rman catalog cmdfile '/export/home/oracle/source/Oracle/scripts/control-M/rman_noarch_rac.cmd'
      
      /export/home/oracle/source/Oracle/scripts/control-M/rman_dbf_bck.sh: [[: not found
      RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 
      Recovery Manager: Release 11.2.0.3.0 - Production on Tue Jan 29 19:00:13 2013
      
      Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
      
      RMAN> connect target *
      2> sql "alter session set optimizer_mode=RULE";
      3> run {
      4> 
      5>      allocate channel t1 type 'sbt_tape';
      6>      allocate channel t2 type 'sbt_tape';
      7>      allocate channel t3 type 'sbt_tape';
      8>      allocate channel t4 type 'sbt_tape';
      9>      send 'NB_ORA_POLICY=RMAN_DATA,NB_ORA_SCHED=Full';
      10> 
      11>      backup as compressed backupset format 'FULL_%u_%d_%s' tag '%TAG' database;
      12>      
      13>      release channel t1;
      14>      release channel t2;
      15>      release channel t3;
      16>      release channel t4;
      17>     }
      18> 
      connected to target database: ***** (DBID=2975797407)
      
      using target database control file instead of recovery catalog
      sql statement: alter session set optimizer_mode=RULE
      
      allocated channel: t1
      channel t1: SID=987 device type=SBT_TAPE
      channel t1: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
      
      allocated channel: t2
      channel t2: SID=1093 device type=SBT_TAPE
      channel t2: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
      
      allocated channel: t3
      channel t3: SID=1388 device type=SBT_TAPE
      channel t3: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
      
      allocated channel: t4
      channel t4: SID=1486 device type=SBT_TAPE
      channel t4: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
      
      sent command to channel: t1
      sent command to channel: t2
      sent command to channel: t3
      sent command to channel: t4
      
      Starting backup at 29-JAN-13
      channel t1: starting compressed full datafile backup set
      channel t1: specifying datafile(s) in backup set
      input datafile file number=00002 name=/opt/u01/oradata/*****/sysaux01.dbf
      input datafile file number=00021 name=/opt/u03/oradata/*****/pm_db01.dbf
      input datafile file number=00018 name=/opt/u03/oradata/*****/pm_indx01.dbf
      input datafile file number=00006 name=/opt/u03/oradata/*****/ems_indx02.dbf
      input datafile file number=00009 name=/opt/u03/oradata/*****/ems_db02.dbf
      input datafile file number=00010 name=/opt/u03/oradata/*****/ems_db03.dbf
      channel t1: starting piece 1 at 29-JAN-13
      channel t2: starting compressed full datafile backup set
      channel t2: specifying datafile(s) in backup set
      input datafile file number=00003 name=/opt/u01/oradata/*****/undotbs01.dbf
      input datafile file number=00020 name=/opt/u04/oradata/*****/pm_db03.dbf
      input datafile file number=00017 name=/opt/u02/oradata/*****/pm_indx03.dbf
      input datafile file number=00005 name=/opt/u02/oradata/*****/ems_indx01.dbf
      input datafile file number=00016 name=/opt/u02/oradata/*****/emsevent_db02.dbf
      input datafile file number=00011 name=/opt/u02/oradata/*****/emsevent_indx03.dbf
      channel t2: starting piece 1 at 29-JAN-13
      channel t3: starting compressed full datafile backup set
      channel t3: specifying datafile(s) in backup set
      input datafile file number=00022 name=/opt/u02/oradata/*****/pm_db02.dbf
      input datafile file number=00019 name=/opt/u04/oradata/*****/pm_indx02.dbf
      input datafile file number=00008 name=/opt/u04/oradata/*****/ems_db01.dbf
      input datafile file number=00012 name=/opt/u04/oradata/*****/emsevent_indx01.dbf
      input datafile file number=00015 name=/opt/u04/oradata/*****/emsevent_db01.dbf
      input datafile file number=00007 name=/opt/u04/oradata/*****/ems_indx03.dbf
      channel t3: starting piece 1 at 29-JAN-13
      channel t4: starting compressed full datafile backup set
      channel t4: specifying datafile(s) in backup set
      input datafile file number=00001 name=/opt/u01/oradata/*****/system01.dbf
      input datafile file number=00004 name=/opt/u01/oradata/*****/users01.dbf
      input datafile file number=00013 name=/opt/u03/oradata/*****/emsevent_indx02.dbf
      input datafile file number=00014 name=/opt/u03/oradata/*****/emsevent_db03.dbf
      channel t4: starting piece 1 at 29-JAN-13
      *RMAN-03009: failure of backup command on t3 channel at 01/29/2013 19:01:36*
      *ORA-19506: failed to create sequential file, name="FULL_48o0lcqe_*****_3208", parms=""*
      *ORA-27028: skgfqcre: sbtbackup returned error*
      *ORA-19511: Error received from media manager layer, error text:*
         *VxBSACreateObject: Failed with error:*
         *Server Status:  problems encountered during setup of shared memory*
      *channel t3 disabled, job failed on it will be run on another channel*
      *RMAN-03009: failure of backup command on t4 channel at 01/29/2013 19:01:36*
      *ORA-19506: failed to create sequential file, name="FULL_49o0lcqf_*****_3209", parms=""*
      *ORA-27028: skgfqcre: sbtbackup returned error*
      *ORA-19511: Error received from media manager layer, error text:*
         *VxBSACreateObject: Failed with error:*
         *Server Status:  problems encountered during setup of shared memory*
      *channel t4 disabled, job failed on it will be run on another channel*
      channel t2: finished piece 1 at 29-JAN-13
      piece handle=FULL_47o0lcqe_*****_3207 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t2: backup set complete, elapsed time: 01:59:23
      channel t2: starting compressed full datafile backup set
      channel t2: specifying datafile(s) in backup set
      input datafile file number=00022 name=/opt/u02/oradata/*****/pm_db02.dbf
      input datafile file number=00019 name=/opt/u04/oradata/*****/pm_indx02.dbf
      input datafile file number=00008 name=/opt/u04/oradata/*****/ems_db01.dbf
      input datafile file number=00012 name=/opt/u04/oradata/*****/emsevent_indx01.dbf
      input datafile file number=00015 name=/opt/u04/oradata/*****/emsevent_db01.dbf
      input datafile file number=00007 name=/opt/u04/oradata/*****/ems_indx03.dbf
      channel t2: starting piece 1 at 29-JAN-13
      channel t1: finished piece 1 at 29-JAN-13
      piece handle=FULL_46o0lcqe_*****_3206 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t1: backup set complete, elapsed time: 03:13:51
      channel t1: starting compressed full datafile backup set
      channel t1: specifying datafile(s) in backup set
      input datafile file number=00001 name=/opt/u01/oradata/*****/system01.dbf
      input datafile file number=00004 name=/opt/u01/oradata/*****/users01.dbf
      input datafile file number=00013 name=/opt/u03/oradata/*****/emsevent_indx02.dbf
      input datafile file number=00014 name=/opt/u03/oradata/*****/emsevent_db03.dbf
      channel t1: starting piece 1 at 29-JAN-13
      channel t2: finished piece 1 at 29-JAN-13
      piece handle=FULL_4ao0ljqa_*****_3210 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t2: backup set complete, elapsed time: 01:20:53
      channel t2: starting compressed full datafile backup set
      channel t2: specifying datafile(s) in backup set
      including current control file in backup set
      channel t2: starting piece 1 at 29-JAN-13
      channel t2: finished piece 1 at 29-JAN-13
      piece handle=FULL_4co0loi0_*****_3212 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t2: backup set complete, elapsed time: 00:00:25
      channel t2: starting compressed full datafile backup set
      channel t2: specifying datafile(s) in backup set
      including current SPFILE in backup set
      channel t2: starting piece 1 at 29-JAN-13
      channel t2: finished piece 1 at 29-JAN-13
      piece handle=FULL_4do0loj1_*****_3213 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t2: backup set complete, elapsed time: 00:00:25
      channel t1: finished piece 1 at 29-JAN-13
      piece handle=FULL_4bo0lo5u_*****_3211 tag=%TAG comment=API Version 2.0,MMS Version 5.0.0.0
      channel t1: backup set complete, elapsed time: 00:11:24
      Finished backup at 29-JAN-13
      
      released channel: t1
      
      released channel: t2
      
      released channel: t3
      *released channel: t4*
      *RMAN-00571: ===========================================================*
      *RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============*
      *RMAN-00571: ===========================================================*
      *RMAN-03002: failure of release command at 01/29/2013 22:25:47*
      *RMAN-06012: channel: t3 not allocated*
      
      Recovery Manager complete.
      
      no rows selected
      
      *ORA-27028: skgfqcre: sbtbackup returned error*
      *ORA-19511: Error received from media manager layer, error text:*
         *VxBSACreateObject: Failed with error:*
      *ORA-27028: skgfqcre: sbtbackup returned error*
      *ORA-19511: Error received from media manager layer, error text:*
         *VxBSACreateObject: Failed with error:*
      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
      
      Tue Jan 29 22:25:53 EST 2013 ----------------End of Script------------------
      
      exit 0
      
      
      Remote Host connection details
      ==============================
      Control-M/Agent: ********.corp.*****.com
      Protocol: SSH
      SSH Port: 00022
      SSH Algorithm: BLOWFISH
      Compression: N
        • 1. Re: RMAN backup failure
          clementeOTN
          Hi
          It seems you have a problem with netbackup integration, especially with drives t3 and t4 .
          Have you seen any aplication/operating system error in the netbackup server ?

          Errors must be readed from "down to up",so, it seems that RMAN tells you he had a memory problem reported by netbackup ( Error received from media manager layer,)

          oooo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo
          RMAN-03009: failure of backup command on t3 channel at 01/29/2013 19:01:36
          ORA-19506: failed to create sequential file, name="FULL_48o0lcqe_****_3208", parms=""*
          *ORA-27028: skgfqcre: sbtbackup returned error*
          *ORA-19511: Error received from media manager layer, error text:*
          *VxBSACreateObject: Failed with error:*
          *Server Status: problems encountered during setup of shared memory*

          --oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo--oo
          • 2. Re: RMAN backup failure
            GeetaM
            Thank you for your quick response, let me check with my netbackup team and see what they have to say on it.
            • 3. Re: RMAN backup failure
              652072
              Hello, This is a shared memory issue.. Increasing the shared memory should fix the problem.... Contact your Tape Library Vendor Team...

              Regards,
              Naga

              Edited by: nagnrik on Jan 31, 2013 5:39 PM