1 2 Previous Next 17 Replies Latest reply: Dec 13, 2012 2:45 AM by user13420250 RSS

    Controlfile not present in ASM

    user13420250
      Hi,

      in our backup logfile we see the follwoing error (db 11.2.0.1.0 64 bit)

      RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 12/10/2012 01:46:31
      ORA-19504: failed to create file "+FRA/ngp01/controlfile/c-1446976194-20121210-02"
      ORA-17502: ksfdcre:4 Failed to create file +FRA/ngp01/controlfile/c-1446976194-20121210-02
      ORA-15173: entry 'ngp01' does not exist in directory '/'

      I have checked and the ngp01 directory does not exist in ASM.
      But whats also strange..
      When checking the v$controlfile, I see the following:

      SQL> select name from v$controlfile;

      NAME
      --------------------------------------------------------------------------------
      +FRA/ngp01/controlfile/current.3434.796827271


      Same directory that does not exist according to RMAN and also can't be seen by me using asmcmd.
      So, how come the database is still running ?
      I was also able to create a backup to trace from this controlfile, and the backup looks good. Everything mentioned in the file is correct.
      So, it seems the db thinks the controlfile is still available, but it's not !
      Could it be, the controlfile is cached by ASM or something like that ? And what is the db updating if there is no controlfile to update ?
      There are no errors in the alert.log of the database.
      Anyone who has seen this before ?

      Thnx !
        • 1. Re: Controlfile not present in ASM
          Levi Pereira
          Hi,

          Post here output of:
          RMAN > SHOW ALL;
          
          SQL> SHOW PARAMETER db_recovery_file
          Please, use text formatting using
           tag at begin and end of output.                                                                                                                                                                                                                                                                                                                                                                        
          • 2. Re: Controlfile not present in ASM
            71617
            hi,

            look like you try to backup your controlfile to ASM.
            Try run
            RMAN> backup current controlfile;

            If you have tape library setup right, you should see it backup current controlfile to tape.

            p.s. You only have one control file in your database, you should create second on in different location.
            • 3. Re: Controlfile not present in ASM
              user13420250
              Actually, we have two control files on ASM storage.
              I only showd one in my guestion.
              But both files are not where the db thinks they should be.
              The backup of the controlfile is not really the main issue.
              I don't understand how the database can run normal, when no control file is present ?
              This is from v$controlfile


              SQL> select name from v$controlfile;

              NAME
              --------------------------------------------------------------------------------
              +NGPDATA/ngp01/controlfile/current.263.796827271
              +FRA/ngp01/controlfile/current.3434.796827271

              But both files are not there...
              • 4. Re: Controlfile not present in ASM
                user13420250
                RMAN> show all;

                RMAN configuration parameters for database with db_unique_name NGP01I are:
                CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
                CONFIGURE BACKUP OPTIMIZATION ON;
                CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
                CONFIGURE CONTROLFILE AUTOBACKUP ON;
                CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
                CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
                CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
                CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
                CONFIGURE MAXSETSIZE TO UNLIMITED; # default
                CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
                CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
                CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
                CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
                CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/usr/local/oracle/11.2.0/dbs/snapcf_NGP01i4.f'; # default
                • 5. Re: Controlfile not present in ASM
                  Levi Pereira
                  Hi,

                  This not match:
                  ORA-17502: ksfdcre:4 Failed to create file +FRA/ngp01/controlfile/c-1446976194-20121210-02
                  
                  CONFIGURE CONTROLFILE AUTOBACKUP ON;
                  CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
                  See this example:
                  SQL> show parameter db_recover
                  
                  NAME                                 TYPE        VALUE
                  ------------------------------------ ----------- ------------------------------
                  db_recovery_file_dest                string      +FRA
                  db_recovery_file_dest_size           big integer 4977M
                  
                  
                  RMAN> show all;
                  CONFIGURE CONTROLFILE AUTOBACKUP ON;
                  CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; 
                  
                  RMAN> backup current controlfile;
                  
                  Starting backup at 11-DEC-12
                  using channel ORA_DISK_1
                  using channel ORA_DISK_2
                  channel ORA_DISK_1: starting compressed full datafile backup set
                  channel ORA_DISK_1: specifying datafile(s) in backup set
                  including current control file in backup set
                  channel ORA_DISK_1: starting piece 1 at 11-DEC-12
                  channel ORA_DISK_1: finished piece 1 at 11-DEC-12
                  piece handle=+FRA/prod/backupset/2012_12_11/ncnnf0_tag20121211t102929_0.278.8
                  channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
                  Finished backup at 11-DEC-12
                  
                  Starting Control File and SPFILE Autobackup at 11-DEC-12
                  piece handle=+FRA/prod/autobackup/2012_12_11/s_801743371.275.801743371 commen
                  Finished Control File and SPFILE Autobackup at 11-DEC-12
                  The directory structure to store AUTOBACKUP using FRA is

                  <DG_NAME>/<DB_UNIQUE_NAME>/AUTOBACKUP/<DATE>/FILE_NAME

                  Can you reproduce this test above on your database?


                  Levi Pereira
                  • 6. Re: Controlfile not present in ASM
                    user13420250
                    Thnx levi !

                    But the backup is no longer a problem.
                    That part we already solved. The question is now, how the database can continue to run without controlfiles.
                    The current controlfiles are not were they are supposed to be.
                    I was wondering if anybody has experienced this same issue. A database that keeps running normal without its controlfiles.
                    • 7. Re: Controlfile not present in ASM
                      Levi Pereira
                      user13420250 wrote:
                      Thnx levi !

                      But the backup is no longer a problem.
                      That part we already solved. The question is now, how the database can continue to run without controlfiles.
                      The current controlfiles are not were they are supposed to be.
                      I was wondering if anybody has experienced this same issue. A database that keeps running normal without its controlfiles.
                      I think you are confused about the error.

                      The error is caused due RMAN was unable to create AUTOBACKUP (due ORA-15173) ,not because CONTROLFILE is missing.

                      See the filename.
                      c-1446976194-20121210-02 - AUTOBACKUP File.
                      current.3434.796827271 - Controlfile

                      ORA-19504: failed to create file "+FRA/ngp01/controlfile/c-1446976194-20121210-02"
                      ORA-17502: ksfdcre:4 Failed to create file +FRA/ngp01/controlfile/c-1446976194-20121210-02
                      ORA-15173: entry 'ngp01' does not exist in directory '/'
                      
                      SQL> select name from v$controlfile;
                      NAME
                      +FRA/ngp01/controlfile/current.3434.796827271
                      Database abort if any failure occur on Controlfile.

                      Regards,
                      Levi Pereira
                      • 8. Re: Controlfile not present in ASM
                        user13420250
                        At first I notived the backup error, that error was right. the directory mentioned did not exist.
                        But when I was investigating the problem, I noticed the missing controlfiles.

                        The database is expecting the controlfiles to be at the location I posted before (taken from v$controlfile), but they are not there !!
                        I can execute a backup controlfile to trace and the lines in the trace look fine.
                        But there is no controlfile on (ASM)disk.... I don't understand how the database can still run normally without errors.

                        I executed a validate database in rman..

                        channel ORA_DISK_1: starting validation of datafile
                        channel ORA_DISK_1: specifying datafile(s) for validation
                        RMAN-00571: ===========================================================
                        RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                        RMAN-00571: ===========================================================
                        RMAN-03009: failure of validate command on ORA_DISK_1 channel at 12/11/2012 14:23:05
                        ORA-19962: compression algorithm +NGPDATA/ngp01/controlfile/current.263.796827271 is not valid with release
                        ORA-15173: entry 'ngp01' does not exist in directory '/'

                        RMAN cannot find the current controlfile.
                        The directory does not exist..

                        ASMCMD> pwd
                        +
                        ASMCMD> cd NGPDATA
                        ASMCMD> cd ngp01
                        ASMCMD-08002: entry 'ngp01' does not exist in directory '+NGPDATA/'
                        ASMCMD> pwd
                        +NGPDATA

                        So, the database is running without an valid controlfile at this moment.
                        Or am I missing something ?
                        • 9. Re: Controlfile not present in ASM
                          Catch_22
                          How did you manage to loose the controlfile?
                          Are the physical blocks still there?
                          Is the database doing anything?

                          I suspect that a database shutdown or startup and checkpoint will fail.
                          • 10. Re: Controlfile not present in ASM
                            user13420250
                            The database is running for months now (production :( ...)
                            I also think a bounce of the database will cause more problems. That's why we keep it runnin for now.
                            I don't know how the controlfiles got lost, all I can see now is that they are not where they should be
                            Checkpoints are occuring

                            SQL> select CHECKPOINT_TIME from v$datafile_header;

                            CHECKPOIN
                            ---------
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12
                            12-DEC-12

                            8 rows selected.

                            Could it be that the controlfile on disk is not updated by the db ? It looks like the controlfile is in memory or something.
                            • 11. Re: Controlfile not present in ASM
                              user13420250
                              We also see the timestamps are not actual for other databases.
                              Datafiles have an actual timestamp, but the controlfiles have not ? Is that correct ?


                              For example :

                              SQL> select name from v$controlfile
                              2 ;

                              NAME
                              --------------------------------------------------------------------------------
                              +NGPDATA/ngp02i/controlfile/current.282.797615437
                              +FRA/ngp02i/controlfile/current.4010.797615437



                              ASMCMD> ls -l
                              Type Redund Striped Time Sys Name
                              CONTROLFILE HIGH FINE NOV 01 16:00:00 Y Current.282.797615437
                              ASMCMD> pwd
                              +NGPDATA/NGP02I/controlfile                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                              • 12. Re: Controlfile not present in ASM
                                Catch_22
                                According to your previous post it seems you still have one of the multiplexed controlfiles stored in the +NGPDATA diskgroup. You can try to find out the ins and odds of why your database is still working, but I'm afraid it won't fix your problem of missing one of the multiplexed controlfiles.

                                What is the output of the following:

                                <pre>
                                $ asmcmd
                                ASMCMD> cd <DISKGROUP_NAME>
                                ASMCMD> find -t controlfile . *
                                </pre>

                                I suggest to shutdown the database and then use RMAN or ASMCMD to duplicate the controlfile. If you have access to My Oracle support you can see the following notes for detailed information:

                                How to duplicate a controlfile when ASM is involved (345180.1)
                                How To Mirror Controlfile To FRA On ASM (1305674.1)

                                For instance:

                                <pre>
                                SQL> shutdown immediate
                                SQL> startup force nomount
                                RMAN> restore controlfile to 'FRA' from '+NGPDATA/ngp01/controlfile/current.263.796827271'

                                If you need to recreate the ASM directory structure, try the following:

                                SQL> alter diskgroup '+FRA' add directory '+FRA/ngp02i/controlfile';
                                </pre>
                                • 13. Re: Controlfile not present in ASM
                                  user13420250
                                  That controlfile belongs to another database. I posted it to show the timestamps on ASM fot the controlfiles are not updated.
                                  Fot thsi specific db both controlfiles are missing on ASM
                                  • 14. Re: Controlfile not present in ASM
                                    Levi Pereira
                                    I'm confused and and curious.
                                    RMAN-03009: failure of validate command on ORA_DISK_1 channel at 12/11/2012 14:23:05
                                    ORA-19962: compression algorithm +NGPDATA/ngp01/controlfile/current.263.796827271 is not valid with release
                                    ORA-15173: entry 'ngp01' does not exist in directory '/'
                                    
                                    ASMCMD> cd NGPDATA
                                    ASMCMD> cd ngp01
                                    ASMCMD-08002: entry 'ngp01' does not exist in directory '+NGPDATA/'
                                    Please don't mix database ( eg. ngp01 and npg02i)


                                    Can you post here output of (please post all output don't cut):
                                    $ export ORACLE_SID=ngp01
                                    
                                    $ sqlpus / as sysdba
                                    
                                    SQL> show parameter db_recover
                                    
                                    SQL> select open_mode from v$database;
                                    
                                    SQL> select name,status from v$controlfile;
                                    
                                    $ rman target /
                                    
                                    RMAN > show all;
                                    
                                    RMAN> validate current controlfile;
                                    
                                    RMAN> backup current controlfile;
                                    
                                    
                                    $ export ORACLE_HOME=<GRID_HOME>
                                    $ export ORACLE_SID=+ASM
                                    
                                    $ asmcmd lsof |grep controlfile |grep ngp01
                                    1 2 Previous Next