This discussion is archived
9 Replies Latest reply: Jul 31, 2013 5:09 AM by user8873606 RSS

ASM Disk Mount Error

982094 Newbie
Currently Being Moderated
Hi,
I am consistently getting the following error since yesterday while mounting the ASM disk.

1. I created the ASM disks successfully and created the appropriate ASM DiskGroups using asmca utility.
2. I was able to see the diskgroups as mounted from asmcmd tool.
3. All of a sudden, all the diskgroups were dismounted yesterday and I see the following error in the alert logs for all the
diskgroups.
4. This is using ASM11g

As I exhausted all my avenues to fix, I have turned to you for further assistance. Please do the needful.

------------

SQL> /* ASMCMD */ALTER DISKGROUP DataVSPSI MOUNT
NOTE: cache registered group DATAVSPSI number=3 incarn=0x7934bf5d
NOTE: cache began mount (first) of group DATAVSPSI number=3 incarn=0x7934bf5d
NOTE: Assigning number (3,0) to disk (ORCL:D01_VSP_SI)
Thu Dec 27 17:38:29 2012
NOTE: start heartbeating (grp 3)
kfdp_query(DATAVSPSI): 11
kfdp_queryBg(): 11
NOTE: cache opening disk 0 of grp 3: D01_VSP_SI label:D01_VSP_SI
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 3/0x7934BF5D (DATAVSPSI)
NOTE: starting recovery of thread=1 ckpt=2.1 group=3 (DATAVSPSI)
WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
AU:4 disk_offset(bytes):4333568 io_size:122880 operation:Read type:asynchronous
result:I/O error process_id:32162
WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
AU:4 disk_offset(bytes):4202496 io_size:131072 operation:Read type:asynchronous
result:I/O error process_id:32162
ORA-15080: synchronous I/O operation to a disk failed
ERROR: ASM recovery failed to read ACD
NOTE: cache initiating offline of disk 0 group DATAVSPSI
NOTE: process 32162 initiating offline of disk 0.3940831150 (D01_VSP_SI) with mask 0x7e in group 3
WARNING: Disk D01_VSP_SI in mode 0x7f is now being taken offline
NOTE: initiating PST update: grp = 3, dsk = 0/0xeae44fae, mode = 0x15
kfdp_updateDsk(): 12
kfdp_updateDskBg(): 12
ERROR: too many offline disks in PST (grp 3)
WARNING: Disk D01_VSP_SI in mode 0x7f offline aborted
Thu Dec 27 17:38:29 2012
NOTE: halting all I/Os to diskgroup DATAVSPSI
NOTE: crash recovery signalled OER-15130
ERROR: ORA-15130 signalled during mount of diskgroup DATAVSPSI
NOTE: cache dismounting (clean) group 3/0x7934BF5D (DATAVSPSI)
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 3/0x7934BF5D (DATAVSPSI)
...........
-----------------------------------------------

Also, I think the lower level disk is fine as I am able to write to the disk as follows :

[oracle@rmanqa01 trace]$ dd of=/dev/sdp1
Test Test
0+1 records in
0+1 records out
10 bytes (10 B) copied, 4.97435 seconds, 0.0 kB/s
[oracle@rmanqa01 trace]$ id
uid=500(oracle) gid=500(oinstall) groups=6(disk),500(oinstall),501(dba),502(oper),503(asmadmin),504(asmdba),505(asmoper),506(horcm)
[oracle@rmanqa01 trace]$ ls -l /dev/sdp1
brw-rw---- 1 root disk 8, 241 Dec 28 11:59 /dev/sdp1
[oracle@rmanqa01 trace]$

--------------------------------------------------

The oracleasm also lists the disk D01_VSP_SI as follows :

[root@rmanqa01 log]# /etc/init.d/oracleasm listdisks
A01_VSP_SI
ADSK01
ARCH_AMS_SI
D01_VSP_SI
DATA_AMS_SI
DDSK01
DEMO_ARCH
DEMO_DATA
L01_VSP_SI
RDSK01
REDO_AMS_SI
You have new mail in /var/spool/mail/root

-------------------------

[root@rmanqa01 log]# /etc/init.d/oracleasm querydisk D01_VSP_SI
Disk "D01_VSP_SI" is a valid ASM disk
[root@rmanqa01 log]#

-------------------------
kfed tool was giving proper data till yesterday.
Today, I am getting the following :

./kfed read /dev/oracleasm/disks/D01_VSP_SI

kfbh.endian: 84 ; 0x000: 0x54
kfbh.hard: 101 ; 0x001: 0x65
kfbh.type: 115 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 116 ; 0x003: 0x74
kfbh.block.blk: 1936020512 ; 0x004: T=0 NUMB=0x73655420
kfbh.block.obj: 2147486324 ; 0x008: TYPE=0x8 NUMB=0xa74
kfbh.check: 2886846267 ; 0x00c: 0xac11c73b
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message
[oracle@rmanqa01 bin]$
--------------------------------------------------------

Please help.

Thanks

V V

Edited by: user13479556 on Dec 28, 2012 12:00 PM
  • 1. Re: ASM Disk Mount Error
    rp0428 Guru
    Currently Being Moderated
    Welcome to the forum!

    You might get better help if you mark this question ANSWERED and post it in the ASM forum.
    Automatic Storage Management
  • 2. Re: ASM Disk Mount Error
    982094 Newbie
    Currently Being Moderated
    Thanks
  • 3. Re: ASM Disk Mount Error
    982094 Newbie
    Currently Being Moderated
    Where is the answer ? The status shows as "answered"....
  • 4. Re: ASM Disk Mount Error
    982094 Newbie
    Currently Being Moderated
    Where is the answer ? So markiing it as "un-answered"
  • 5. Re: ASM Disk Mount Error
    982094 Newbie
    Currently Being Moderated
    Anybody any clue/idea as to what's going on here ?
    Please help.
  • 6. Re: ASM Disk Mount Error
    berx Explorer
    Currently Being Moderated
    I'm not sure what the problem of your configuration is, but after your
    [oracle@rmanqa01 trace]$ dd of=/dev/sdp1
    Test Test
    0+1 records in
    0+1 records out
    10 bytes (10 B) copied, 4.97435 seconds, 0.0 kB/s
    [oracle@rmanqa01 trace]$ id
    uid=500(oracle) gid=500(oinstall) groups=6(disk),500(oinstall),501(dba),502(oper),503(asmadmin),504(asmdba),505(asmoper),506(horcm)
    [oracle@rmanqa01 trace]$ ls -l /dev/sdp1
    brw-rw---- 1 root disk 8, 241 Dec 28 11:59 /dev/sdp1
    I'm quite sure you damaged the disk.
  • 7. Re: ASM Disk Mount Error
    982094 Newbie
    Currently Being Moderated
    Thanks Berx for pointing that out.

    I deleted the ASM disks and the diskgroups and re-created a fresh and was able to start the ASM instance and mount the DGs.

    Now the mount is persistent, but the dbca fails with the following error in the alert log. Under what situations can this error be seen ?

    -----------
    NOTE: Loaded library: System
    SUCCESS: diskgroup DATAAMSSI was mounted
    SUCCESS: diskgroup ARCHAMSSI was mounted
    ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.DATAAMSSI.dg*ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.ARCHAMSSI.dg*Mon Dec 31 20:25:25 2012
    SUCCESS: diskgroup LOGAMSSI was mounted
    Mon Dec 31 20:25:25 2012
    ERROR: failed to establish dependency between database R3AMSSI and diskgroup resource ora.LOGAMSSI.dg
    Mon Dec 31 20:25:25 2012
    Successful mount of redo thread 1, with mount id 784925673
    Completed: Create controlfile reuse set database "R3AMSSI"
    MAXINSTANCES 8
    MAXLOGHISTORY 1
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    Datafile
    '+DATAAMSSI/R3AMSSI/system01.dbf',
    '+DATAAMSSI/R3AMSSI/sysaux01.dbf',
    '+DATAAMSSI/R3AMSSI/undotbs01.dbf',
    '+DATAAMSSI/R3AMSSI/users01.dbf'
    LOGFILE GROUP 1 ('+LOGAMSSI/R3AMSSI/redo01.log') SIZE 51200K,
    GROUP 2 ('+LOGAMSSI/R3AMSSI/redo02.log') SIZE 51200K,
    GROUP 3 ('+LOGAMSSI/R3AMSSI/redo03.log') SIZE 51200K RESETLOGS
    Stopping background process MMNL
    Stopping background process MMON
    Starting background process MMON
    Starting background process MMNL
    Mon Dec 31 20:25:28 2012
    MMON started with pid=17, OS id=10452
    ALTER SYSTEM enable restricted session;
    Mon Dec 31 20:25:28 2012
    MMNL started with pid=18, OS id=10454
    alter database "R3AMSSI" open resetlogs
    RESETLOGS after incomplete recovery UNTIL CHANGE 945183
    Errors in file /u01/app/oracle/diag/rdbms/r3amssi/R3AMSSI/trace/R3AMSSI_ora_10434.trc:
    ORA-00313: open failed for members of log group 1 of thread 1
    ORA-00312: online log 1 thread 1: '+LOGAMSSI/r3amssi/redo01.log'
    ORA-17503: ksfdopn:2 Failed to open file +LOGAMSSI/r3amssi/redo01.log
    ORA-15173: entry 'redo01.log' does not exist in directory 'r3amssi'
    Errors in file /u01/app/oracle/diag/rdbms/r3amssi/R3AMSSI/trace/R3AMSSI_ora_10434.trc:
    ORA-00313: open failed for members of log group 1 of thread 1
    ORA-00312: online log 1 thread 1: '+LOGAMSSI/r3amssi/redo01.log'
    ORA-17503: ksfdopn:2 Failed to open file +LOGAMSSI/r3amssi/redo01.log
    ORA-15173: entry 'redo01.log' does not exist in directory 'r3amssi'
    Mon Dec 31 20:25:29 2012
    Checker run found 5 new persistent data failures
    Mon Dec 31 20:27:07 2012
    .................
    ----------------------------------

    I can verify from asmcmd that the redologs mentioned above as missing is very much present in '+LOGAMSSI/R3AMSSI'

    Thanks

    V V
  • 8. Re: ASM Disk Mount Error
    berx Explorer
    Currently Being Moderated
    I personally have no clue right now.
    do you have any additional informations in R3AMSSI_ora_10434.trc ?

    best regards,
    Martin
  • 9. Re: ASM Disk Mount Error
    user8873606 Newbie
    Currently Being Moderated


    Try this entry on MOS

     

     

    This error is harmless.
    Documented in Bug 8304720
    Abstract: ERROR: FAILED TO ESTABLISH DEPENDENCY BETWEEN DB & DISKGROUP
    This is an internal BUG.

    The db-dg dependency will be created later on if the diskgroup is used by the db again; the dependency can also be created manually through cmdline tool.

    Automatically it will resolve itself later on when the db tries to use the diskgroup. Or we can manually establish the dependancy through any command line utility.

     

    For establishing dependencies between database and diskgroups, please use DB home owner running srvctl from DB home with

    $ srvctl modify database -d <db_unique_name> -a "<diskgroup_list>"

    For example:
    $ srvctl modify database -d db112 -a "DATA,FRA"

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points