9 Replies Latest reply: Jul 31, 2013 7:09 AM by user8873606 RSS

    ASM Disk Mount Error

    982094
      Hi,
      I am consistently getting the following error since yesterday while mounting the ASM disk.

      1. I created the ASM disks successfully and created the appropriate ASM DiskGroups using asmca utility.
      2. I was able to see the diskgroups as mounted from asmcmd tool.
      3. All of a sudden, all the diskgroups were dismounted yesterday and I see the following error in the alert logs for all the
      diskgroups.
      4. This is using ASM11g

      As I exhausted all my avenues to fix, I have turned to you for further assistance. Please do the needful.

      ------------

      SQL> /* ASMCMD */ALTER DISKGROUP DataVSPSI MOUNT
      NOTE: cache registered group DATAVSPSI number=3 incarn=0x7934bf5d
      NOTE: cache began mount (first) of group DATAVSPSI number=3 incarn=0x7934bf5d
      NOTE: Assigning number (3,0) to disk (ORCL:D01_VSP_SI)
      Thu Dec 27 17:38:29 2012
      NOTE: start heartbeating (grp 3)
      kfdp_query(DATAVSPSI): 11
      kfdp_queryBg(): 11
      NOTE: cache opening disk 0 of grp 3: D01_VSP_SI label:D01_VSP_SI
      NOTE: F1X0 found on disk 0 au 2 fcn 0.0
      NOTE: cache mounting (first) external redundancy group 3/0x7934BF5D (DATAVSPSI)
      NOTE: starting recovery of thread=1 ckpt=2.1 group=3 (DATAVSPSI)
      WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
      AU:4 disk_offset(bytes):4333568 io_size:122880 operation:Read type:asynchronous
      result:I/O error process_id:32162
      WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
      AU:4 disk_offset(bytes):4202496 io_size:131072 operation:Read type:asynchronous
      result:I/O error process_id:32162
      ORA-15080: synchronous I/O operation to a disk failed
      ERROR: ASM recovery failed to read ACD
      NOTE: cache initiating offline of disk 0 group DATAVSPSI
      NOTE: process 32162 initiating offline of disk 0.3940831150 (D01_VSP_SI) with mask 0x7e in group 3
      WARNING: Disk D01_VSP_SI in mode 0x7f is now being taken offline
      NOTE: initiating PST update: grp = 3, dsk = 0/0xeae44fae, mode = 0x15
      kfdp_updateDsk(): 12
      kfdp_updateDskBg(): 12
      ERROR: too many offline disks in PST (grp 3)
      WARNING: Disk D01_VSP_SI in mode 0x7f offline aborted
      Thu Dec 27 17:38:29 2012
      NOTE: halting all I/Os to diskgroup DATAVSPSI
      NOTE: crash recovery signalled OER-15130
      ERROR: ORA-15130 signalled during mount of diskgroup DATAVSPSI
      NOTE: cache dismounting (clean) group 3/0x7934BF5D (DATAVSPSI)
      NOTE: lgwr not being msg'd to dismount
      NOTE: cache dismounted group 3/0x7934BF5D (DATAVSPSI)
      ...........
      -----------------------------------------------

      Also, I think the lower level disk is fine as I am able to write to the disk as follows :

      [oracle@rmanqa01 trace]$ dd of=/dev/sdp1
      Test Test
      0+1 records in
      0+1 records out
      10 bytes (10 B) copied, 4.97435 seconds, 0.0 kB/s
      [oracle@rmanqa01 trace]$ id
      uid=500(oracle) gid=500(oinstall) groups=6(disk),500(oinstall),501(dba),502(oper),503(asmadmin),504(asmdba),505(asmoper),506(horcm)
      [oracle@rmanqa01 trace]$ ls -l /dev/sdp1
      brw-rw---- 1 root disk 8, 241 Dec 28 11:59 /dev/sdp1
      [oracle@rmanqa01 trace]$

      --------------------------------------------------

      The oracleasm also lists the disk D01_VSP_SI as follows :

      [root@rmanqa01 log]# /etc/init.d/oracleasm listdisks
      A01_VSP_SI
      ADSK01
      ARCH_AMS_SI
      D01_VSP_SI
      DATA_AMS_SI
      DDSK01
      DEMO_ARCH
      DEMO_DATA
      L01_VSP_SI
      RDSK01
      REDO_AMS_SI
      You have new mail in /var/spool/mail/root

      -------------------------

      [root@rmanqa01 log]# /etc/init.d/oracleasm querydisk D01_VSP_SI
      Disk "D01_VSP_SI" is a valid ASM disk
      [root@rmanqa01 log]#

      -------------------------
      kfed tool was giving proper data till yesterday.
      Today, I am getting the following :

      ./kfed read /dev/oracleasm/disks/D01_VSP_SI

      kfbh.endian: 84 ; 0x000: 0x54
      kfbh.hard: 101 ; 0x001: 0x65
      kfbh.type: 115 ; 0x002: *** Unknown Enum ***
      kfbh.datfmt: 116 ; 0x003: 0x74
      kfbh.block.blk: 1936020512 ; 0x004: T=0 NUMB=0x73655420
      kfbh.block.obj: 2147486324 ; 0x008: TYPE=0x8 NUMB=0xa74
      kfbh.check: 2886846267 ; 0x00c: 0xac11c73b
      kfbh.fcn.base: 0 ; 0x010: 0x00000000
      kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
      kfbh.spare1: 0 ; 0x018: 0x00000000
      kfbh.spare2: 0 ; 0x01c: 0x00000000
      ERROR!!!, failed to get the oracore error message
      [oracle@rmanqa01 bin]$
      --------------------------------------------------------

      Please help.

      Thanks

      V V

      Edited by: user13479556 on Dec 28, 2012 12:00 PM
        • 1. Re: ASM Disk Mount Error
          rp0428
          Welcome to the forum!

          You might get better help if you mark this question ANSWERED and post it in the ASM forum.
          Automatic Storage Management
          • 2. Re: ASM Disk Mount Error
            982094
            Thanks
            • 3. Re: ASM Disk Mount Error
              982094
              Where is the answer ? The status shows as "answered"....
              • 4. Re: ASM Disk Mount Error
                982094
                Where is the answer ? So markiing it as "un-answered"
                • 5. Re: ASM Disk Mount Error
                  982094
                  Anybody any clue/idea as to what's going on here ?
                  Please help.
                  • 6. Re: ASM Disk Mount Error
                    berx
                    I'm not sure what the problem of your configuration is, but after your
                    [oracle@rmanqa01 trace]$ dd of=/dev/sdp1
                    Test Test
                    0+1 records in
                    0+1 records out
                    10 bytes (10 B) copied, 4.97435 seconds, 0.0 kB/s
                    [oracle@rmanqa01 trace]$ id
                    uid=500(oracle) gid=500(oinstall) groups=6(disk),500(oinstall),501(dba),502(oper),503(asmadmin),504(asmdba),505(asmoper),506(horcm)
                    [oracle@rmanqa01 trace]$ ls -l /dev/sdp1
                    brw-rw---- 1 root disk 8, 241 Dec 28 11:59 /dev/sdp1
                    I'm quite sure you damaged the disk.
                    • 7. Re: ASM Disk Mount Error
                      982094
                      Thanks Berx for pointing that out.

                      I deleted the ASM disks and the diskgroups and re-created a fresh and was able to start the ASM instance and mount the DGs.

                      Now the mount is persistent, but the dbca fails with the following error in the alert log. Under what situations can this error be seen ?

                      -----------
                      NOTE: Loaded library: System
                      SUCCESS: diskgroup DATAAMSSI was mounted
                      SUCCESS: diskgroup ARCHAMSSI was mounted
                      ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.DATAAMSSI.dg*ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.ARCHAMSSI.dg*Mon Dec 31 20:25:25 2012
                      SUCCESS: diskgroup LOGAMSSI was mounted
                      Mon Dec 31 20:25:25 2012
                      ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.LOGAMSSI.dg
                      Mon Dec 31 20:25:25 2012
                      Successful mount of redo thread 1, with mount id 784925673
                      Completed: Create controlfile reuse set database "R3AMSSI"
                      MAXINSTANCES 8
                      MAXLOGHISTORY 1
                      MAXLOGFILES 16
                      MAXLOGMEMBERS 3
                      MAXDATAFILES 100
                      Datafile
                      '+DATAAMSSI/R3AMSSI/system01.dbf',
                      '+DATAAMSSI/R3AMSSI/sysaux01.dbf',
                      '+DATAAMSSI/R3AMSSI/undotbs01.dbf',
                      '+DATAAMSSI/R3AMSSI/users01.dbf'
                      LOGFILE GROUP 1 ('+LOGAMSSI/R3AMSSI/redo01.log') SIZE 51200K,
                      GROUP 2 ('+LOGAMSSI/R3AMSSI/redo02.log') SIZE 51200K,
                      GROUP 3 ('+LOGAMSSI/R3AMSSI/redo03.log') SIZE 51200K RESETLOGS
                      Stopping background process MMNL
                      Stopping background process MMON
                      Starting background process MMON
                      Starting background process MMNL
                      Mon Dec 31 20:25:28 2012
                      MMON started with pid=17, OS id=10452
                      ALTER SYSTEM enable restricted session;
                      Mon Dec 31 20:25:28 2012
                      MMNL started with pid=18, OS id=10454
                      alter database "R3AMSSI" open resetlogs
                      RESETLOGS after incomplete recovery UNTIL CHANGE 945183
                      Errors in file /u01/app/oracle/diag/rdbms/r3amssi/R3AMSSI/trace/R3AMSSI_ora_10434.trc:
                      ORA-00313: open failed for members of log group 1 of thread 1
                      ORA-00312: online log 1 thread 1: '+LOGAMSSI/r3amssi/redo01.log'
                      ORA-17503: ksfdopn:2 Failed to open file +LOGAMSSI/r3amssi/redo01.log
                      ORA-15173: entry 'redo01.log' does not exist in directory 'r3amssi'
                      Errors in file /u01/app/oracle/diag/rdbms/r3amssi/R3AMSSI/trace/R3AMSSI_ora_10434.trc:
                      ORA-00313: open failed for members of log group 1 of thread 1
                      ORA-00312: online log 1 thread 1: '+LOGAMSSI/r3amssi/redo01.log'
                      ORA-17503: ksfdopn:2 Failed to open file +LOGAMSSI/r3amssi/redo01.log
                      ORA-15173: entry 'redo01.log' does not exist in directory 'r3amssi'
                      Mon Dec 31 20:25:29 2012
                      Checker run found 5 new persistent data failures
                      Mon Dec 31 20:27:07 2012
                      .................
                      ----------------------------------

                      I can verify from asmcmd that the redologs mentioned above as missing is very much present in '+LOGAMSSI/R3AMSSI'

                      Thanks

                      V V
                      • 8. Re: ASM Disk Mount Error
                        berx
                        I personally have no clue right now.
                        do you have any additional informations in R3AMSSI_ora_10434.trc ?

                        best regards,
                        Martin
                        • 9. Re: ASM Disk Mount Error
                          user8873606


                          Try this entry on MOS

                           

                           

                          This error is harmless.
                          Documented in Bug 8304720
                          Abstract: ERROR: FAILED TO ESTABLISH DEPENDENCY BETWEEN DB & DISKGROUP
                          This is an internal BUG.

                          The db-dg dependency will be created later on if the diskgroup is used by the db again; the dependency can also be created manually through cmdline tool.

                          Automatically it will resolve itself later on when the db tries to use the diskgroup. Or we can manually establish the dependancy through any command line utility.

                           

                          For establishing dependencies between database and diskgroups, please use DB home owner running srvctl from DB home with

                          $ srvctl modify database -d <db_unique_name> -a "<diskgroup_list>"

                          For example:
                          $ srvctl modify database -d db112 -a "DATA,FRA"