4 Replies Latest reply: Sep 14, 2013 10:11 AM by Ankur ... RSS

    RMAN-00600: internal error, arguments [6000] [] [] [] []

    Ankur ...

      Hi Experts,

       

      While taking a RMAN cold backup for one of my dev database I was getting the above error. The script was working completely fine for almost 6months and this is the first time I have faced an error like this.

       

      =====================================================RMAN_COLD Backup Script ======================================

      location=/backups/RMAN_COLD/TRCDEV/db

       

      .  /mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/TRCDEV_gddevdb1.env

       

      export ORA_NLS10=/mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/nls/data/9idata

       

      PATH=$ORACLE_HOME/bin:.:$PATH; export PATH

      export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'

       

       

      time rman nocatalog log /backups/RMAN_COLD/TRCDEV/log/trcdev_rman_cold_log.log <<++RMAN_END++

       

      connect target /

      CONFIGURE CONTROLFILE AUTOBACKUP CLEAR;

      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK CLEAR;

      CONFIGURE CONTROLFILE AUTOBACKUP ON;

      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '${location}/cf%F';

      shutdown immediate;

      startup mount;

      run {

      allocate channel c0 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c1 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c2 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c3 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c4 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c5 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c6 type disk format '${location}/DF_%d_%U_S%s_T%t';

      allocate channel c7 type disk format '${location}/DF_%d_%U_S%s_T%t';

      backup

      filesperset 50

      database;

      release channel c0;

      release channel c1;

      release channel c2;

      release channel c3;

      release channel c4;

      release channel c5;

      release channel c6;

      release channel c7;

      }

      run {

      allocate channel c0 type disk format '${location}/cf_%d_%U_S%s_T%t';

      backup current controlfile format '${location}/cf_%d_%U_S%s_T%t';

      release channel c0;

      }

      shutdown normal;

      startup;

      exit;

      ++RMAN_END++

      =======================================Script Ends======================================

       

      Contents of RMAN log file:-

       

      Recovery Manager: Release 11.2.0.3.0 - Production on Fri Sep 13 21:12:01 2013

       

      Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

       

      RMAN>

      RMAN>

      connected to target database: TRCDEV (DBID=3630982182)

      using target database control file instead of recovery catalog

       

      ..

      ..

      ..

       

      RMAN>

      RMAN-00571: ===========================================================

      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

      RMAN-00571: ===========================================================

      RMAN-03002: failure of shutdown command at 09/13/2013 22:12:08

      ORA-01013: user requested cancel of current operation

       

      RMAN>

      database is already started

       

      RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21>

      DBGANY:     CMD type=allocate cmdid=1 status=NOT STARTED

      DBGANY:       1 STEPstepid=1 cmdid=1 status=NOT STARTED chid=c0

      DBGANY:           1 TEXTNOD = -- devalloc

      DBGANY:           2 TEXTNOD = declare

      DBGANY:           3 TEXTNOD =   devtype      varchar2(255);

      DBGANY:           4 TEXTNOD =   chid         varchar2(255);

      DBGANY:           5 TEXTNOD =   debug        number := null;

      DBGANY:           6 TEXTNOD =   options      number := null;

      DBGANY:           7 TEXTNOD =   node         varchar2(255);

      DBGANY:           8 TEXTNOD =   maxsize      number;

      DBGANY:           9 TEXTNOD =   kbytes       number := null;

      DBGANY:          10 TEXTNOD =   parallel     binary_integer := null;

      DBGANY:          11 TEXTNOD =   readrate     number := null;

      DBGANY:          12 TEXTNOD =   rate         number := null;

      DBGANY:          13 TEXTNOD =   sendcmd      varchar2(256);

      DBGANY:          14 TEXTNOD =   vendor       varchar2(256);

      DBGANY:          15 TEXTNOD =   israc        boolean;

      DBGANY:          16 TEXTNOD =   instname     varchar2(17);

      DBGANY:          17 TEXTNOD = begin

      DBGANY:          18 TEXTNOD =

      DBGANY:          19 PRMVAL =  chid := 'c0';

      DBGANY:          20 TEXTNOD =   if debug is not null then

      DBGANY:          21 TEXTNOD =     krmicd.execSql(

      DBGANY:          22 TEXTNOD =            'alter session set events ''immediate trace name krb_trace level '

      DBGANY:          23 TEXTNOD =            ||debug||'''');

      DBGANY:          24 TEXTNOD =   end if;

      DBGANY:          25 TEXTNOD =   if options is not null then

      DBGANY:          26 TEXTNOD =     krmicd.execSql(

      DBGANY:          27 TEXTNOD =            'alter session set events ''immediate trace name krb_options level '

      DBGANY:          28 TEXTNOD =            ||options||'''');

      DBGANY:          29 TEXTNOD =   end if;

      DBGANY:          30 TEXTNOD =   devtype := sys.dbms_backup_restore.deviceAllocate( ident => chid,

      DBGANY:          31 TEXTNOD =        node => node,

      DBGANY:          32 PRMVAL = type=>null,dupcnt=>1

      DBGANY:          33 TEXTNOD =  );

      DBGANY:          34 TEXTNOD =   if kbytes is null then

      DBGANY:          35 TEXTNOD =      maxsize := sys.dbms_backup_restore.deviceQuery

      DBGANY:          36 TEXTNOD =        (sys.dbms_backup_restore.DEVICEQUERY_MAXSIZE);

      DBGANY:          37 TEXTNOD =   else

      DBGANY:          38 TEXTNOD =      maxsize := kbytes;

      DBGANY:          39 TEXTNOD =   end if;

      DBGANY:      40 TEXTNOD =   if maxsize > 0 then
      DBGANY:      41 TEXTNOD =  sys.dbms_backup_restore.setlimit
      DBGANY:      42 TEXTNOD =    (sys.dbms_backup_restore.kbytes, maxsize);
      DBGANY:      43 TEXTNOD =   end if;

       

      DBGANY:      44 TEXTNOD =   krmicd.setChannelInfo
      DBGANY:      45 TEXTNOD = (devtype,
      DBGANY:      46 TEXTNOD =  node,
      DBGANY:      47 TEXTNOD =  maxsize,
      DBGANY:      48 TEXTNOD =  sys.dbms_backup_restore.deviceQuery
      DBGANY:      49 TEXTNOD =    (sys.dbms_backup_restore.DEVICEQUERY_PROXY),
      DBGANY:      50 TEXTNOD =  sys.dbms_backup_restore.deviceQuery
      DBGANY:      51 TEXTNOD =    (sys.dbms_backup_restore.DEVICEQUERY_MAXPROXY));
      DBGANY:      52 TEXTNOD =   if parallel is not null then
      DBGANY:      53 TEXTNOD =  sys.dbms_backup_restore.setlimit
      DBGANY:      54 TEXTNOD =    (sys.dbms_backup_restore.parallel, parallel);
      DBGANY:      55 TEXTNOD =   end if;
      DBGANY:      56 TEXTNOD =   if readrate is not null then
      DBGANY:      57 TEXTNOD =  sys.dbms_backup_restore.setlimit
      DBGANY:      58 TEXTNOD =    (sys.dbms_backup_restore.readrate, readrate);
      DBGANY:      59 TEXTNOD =   end if;
      DBGANY:      60 TEXTNOD =   if rate is not null then
      DBGANY:      61 TEXTNOD =  sys.dbms_backup_restore.setlimit
      DBGANY:      62 TEXTNOD =    (sys.dbms_backup_restore.max_read_kbytes, rate);
      DBGANY:      63 TEXTNOD =   end if;
      DBGANY:      64 TEXTNOD =   if sendcmd is not null then
      DBGANY:      65 TEXTNOD =  sys.dbms_backup_restore.devicecommand(sendcmd, NULL);
      DBGANY:      66 TEXTNOD =   end if;
      DBGANY:      67 TEXTNOD =   krmicd.writeMsg(8030, chid);
      DBGANY:      68 TEXTNOD =   krmicd.getInstance(instname, israc);
      DBGANY:      69 TEXTNOD =   if (israc) then
      DBGANY:      70 TEXTNOD =  krmicd.writeMsg(8605, chid, to_char(krmicd.getSid),
      DBGANY:      71 TEXTNOD =                  to_char(instname), devtype);
      DBGANY:      72 TEXTNOD =   else
      DBGANY:      73 TEXTNOD =  krmicd.writeMsg(8500, chid, to_char(krmicd.getSid), devtype);
      DBGANY:      74 TEXTNOD =   end if;
      DBGANY:      75 TEXTNOD =   vendor := sys.dbms_backup_restore.deviceQuery
      DBGANY:      76 TEXTNOD = (sys.dbms_backup_restore.DEVICEQUERY_VENDOR);
      DBGANY:      77 TEXTNOD =   if vendor is not null then
      DBGANY:      78 TEXTNOD =  krmicd.writemsg(8526, chid, vendor);
      DBGANY:      79 TEXTNOD =   end if;
      DBGANY:      80 TEXTNOD = end;

      RMAN-00571: ===========================================================

      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

      RMAN-00571: ===========================================================

      RMAN-00601: fatal error in recovery manager

      RMAN-03004: fatal error during execution of command

      RMAN-00600: internal error, arguments [6000] [] [] [] []

       

       

      Contents of alert log file when backup started:-

      Shutting down instance (immediate)

      Stopping background process SMCO

      Shutting down instance: further logons disabled

      Stopping background process QMNC

      Stopping background process MMNL

      Stopping background process MMON

      License high water mark = 116

      Fri Sep 13 21:17:07 2013

      Active call for process 1187 user 'oraDEV' program 'oracle@gddevdb1.tireco.com'

      SHUTDOWN: waiting for active calls to complete.

      Fri Sep 13 21:20:13 2013

      Incremental checkpoint up to RBA [0x51.1d8e.0], current log tail at RBA [0x51.1d9a.0]

      Fri Sep 13 21:40:15 2013

      Incremental checkpoint up to RBA [0x51.1df0.0], current log tail at RBA [0x51.1df0.0]

      Fri Sep 13 22:00:17 2013

      Incremental checkpoint up to RBA [0x51.1e18.0], current log tail at RBA [0x51.1e2a.0]

      Fri Sep 13 22:12:08 2013

      Active call for process 1187 user 'oraDEV' program 'oracle@gddevdb1.tireco.com'

      SHUTDOWN: Active sessions prevent database close operation

                Shutdown did not complete in 60 minutes

      Instance shutdown cancelled

      alter database mount

      ORA-1100 signalled during: alter database mount...

      Fri Sep 13 22:20:20 2013

       

      Please let me know how do I troubleshoot the root cause for this error. I checked metalink and found a note which says to "resync catalog" and I did that but it hasn't helped in my case. Please help a quick response will be highly appreciated.

       

      -Thanks