This discussion is archived
5 Replies Latest reply: Sep 27, 2012 3:22 AM by 744969 RSS

ASM timeout parameters?

Dana N Newbie
Currently Being Moderated
Oracle: 11.2.0.3 non-RAC (Oracle Restart grid home)
OS: RHEL Server 5.8

Can timeout parameters be set for an ASM instance or for any downstream database instances dependent on an ASM instance? Our storage and sysadmins ran a test (failing over a controller). The Oracle database instance detected a problem reaching +FLASHREC on the NetApp device (it was trying to access a control file evidently). Approximately one second later the database instance decided to terminate itself. The ASM instance remained up. But subsequent checking with ASMCD showed no ASM Diskgroup available using an LS command. After bouncing the ASM instance all was well--the diskgroups reappeared and we were able to restart our database instance. A second seems a bit unforgiving. Can any timeout-related parameters be set on the ASM or ASM-client instance to provide more wiggle room during a controller failover?

##############################################################################
Some errors we encountered from the database instance alert log. Further below are errors from the ASM instance's alert log
##############################################################################

Wed Mar 14 04:00:55 2012
Archived Log entry 89 added for thread 1 sequence 142 ID 0xbb0a69f4 dest 1:
Wed Mar 14 17:51:06 2012
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 120864
Additional information: -1
WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
Wed Mar 14 17:51:06 2012
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>arc2399.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 120864
Additional information: -1
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>arc2399.trc:
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:2 disk:0 AU:59 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group 2 on disk 0 allocation unit 59
Wed Mar 14 17:51:06 2012
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 120864
Additional information: -1
WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 310): terminating the instance due to error 221
Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
ORA-15081: failed to submit an I/O operation to a disk
Wed Mar 14 17:51:07 2012
License high water mark = 8
Instance terminated by CKPT, pid = 310
USER (ospid: 24054): terminating the instance
Instance terminated by USER, pid = 24054


#######################################
Some errors we encountered from the ASM instance alert log
#######################################

Mon Mar 12 14:56:18 2012
NOTE: ASMB process exiting due to lack of ASM file activity for 305 seconds
Wed Mar 14 17:51:06 2012
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
WARNING: Write Failed. group:1 disk:0 AU:1 offset:1044480 size:4096
WARNING: Hbeat write to PST disk 0.3916384140 (DATAFILE_0000) in group 1 failed.
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
WARNING: Write Failed. group:2 disk:0 AU:1 offset:1044480 size:4096
WARNING: Hbeat write to PST disk 0.3916384141 (FLASHREC_0000) in group 2 failed.
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
WARNING: Write Failed. group:3 disk:0 AU:1 offset:1044480 size:4096
WARNING: Hbeat write to PST disk 0.3916384142 (TEMPFILE_0000) in group 3 failed.
Wed Mar 14 17:51:06 2012
NOTE: process b000+asm (23739) initiating offline of disk 0.3916384140 (DATAFILE_0000) with mask 0x7e in group 1
WARNING: Disk 0 (DATAFILE_0000) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xe96f478c, mask = 0x6a, op = clear
Wed Mar 14 17:51:06 2012
NOTE: process b001+asm (23753) initiating offline of disk 0.3916384141 (FLASHREC_0000) with mask 0x7e in group 2
WARNING: Disk 0 (FLASHREC_0000) in group 2 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 2, dsk = 0/0xe96f478d, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 13 for pid 20, osid 23739
ERROR: no read quorum in group: required 1, found 0 disks
Wed Mar 14 17:51:06 2012
NOTE: process b002+asm (23791) initiating offline of disk 0.3916384142 (TEMPFILE_0000) with mask 0x7e in group 3
WARNING: Disk 0 (TEMPFILE_0000) in group 3 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 3, dsk = 0/0xe96f478e, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 14 for pid 23, osid 23753
ERROR: no read quorum in group: required 1, found 0 disks
Wed Mar 14 17:51:06 2012
NOTE: cache dismounting (not clean) group 1/0x7FAFB779 (DATAFILE)
NOTE: messaging CKPT to quiesce pins Unix process pid: 23826, image: oracle@dot-oraprd04 (B003)
Wed Mar 14 17:51:06 2012
NOTE: halting all I/Os to diskgroup 1 (DATAFILE)
Wed Mar 14 17:51:06 2012
NOTE: LGWR doing non-clean dismount of group 1 (DATAFILE)
NOTE: LGWR sync ABA=6.6196 last written ABA 6.6196
NOTE: cache dismounted group 1/0x7FAFB779 (DATAFILE)
SQL> alter diskgroup DATAFILE dismount force /* ASM SERVER */
Wed Mar 14 17:51:06 2012
NOTE: cache dismounting (not clean) group 2/0x7FAFB77A (FLASHREC)
NOTE: messaging CKPT to quiesce pins Unix process pid: 23836, image: oracle@dot-oraprd04 (B004)
NOTE: halting all I/Os to diskgroup 2 (FLASHREC)
NOTE: LGWR doing non-clean dismount of group 2 (FLASHREC)
NOTE: LGWR sync ABA=5.1120 last written ABA 5.1120
GMON updating disk modes for group 3 at 15 for pid 25, osid 23791
ERROR: no read quorum in group: required 1, found 0 disks
NOTE: cache dismounted group 2/0x7FAFB77A (FLASHREC)
SQL> alter diskgroup FLASHREC dismount force /* ASM SERVER */
Wed Mar 14 17:51:06 2012
NOTE: cache dismounting (not clean) group 3/0x7FAFB77B (TEMPFILE)
NOTE: messaging CKPT to quiesce pins Unix process pid: 23838, image: oracle@dot-oraprd04 (B005)
NOTE: halting all I/Os to diskgroup 3 (TEMPFILE)
NOTE: LGWR doing non-clean dismount of group 3 (TEMPFILE)
NOTE: LGWR sync ABA=6.11 last written ABA 6.11
NOTE: cache dismounted group 3/0x7FAFB77B (TEMPFILE)
SQL> alter diskgroup TEMPFILE dismount force /* ASM SERVER */
WARNING: Offline of disk 0 (TEMPFILE_0000) in group 3 and mode 0x7f failed on ASM inst 1
WARNING: Offline of disk 0 (DATAFILE_0000) in group 1 and mode 0x7f failed on ASM inst 1
WARNING: Offline of disk 0 (FLASHREC_0000) in group 2 and mode 0x7f failed on ASM inst 1
Wed Mar 14 17:51:07 2012
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
Wed Mar 14 17:51:07 2012
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
Wed Mar 14 17:51:07 2012
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ASM Health Checker found 1 new failures
ASM Health Checker found 1 new failures
ASM Health Checker found 1 new failures
Wed Mar 14 17:51:07 2012
NOTE: ASM client <instance name here>:<instance name here> disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_322.trc
Wed Mar 14 17:51:07 2012
NOTE: cache deleting context for group FLASHREC 2/0x7fafb77a
NOTE: cache deleting context for group TEMPFILE 3/0x7fafb77b
NOTE: cache deleting context for group DATAFILE 1/0x7fafb779
GMON dismounting group 2 at 16 for pid 27, osid 23836
GMON dismounting group 1 at 17 for pid 26, osid 23826
NOTE: Disk in mode 0x8 marked for de-assignment
GMON dismounting group 3 at 18 for pid 28, osid 23838
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
SUCCESS: diskgroup FLASHREC was dismounted
SUCCESS: alter diskgroup FLASHREC dismount force /* ASM SERVER */
SUCCESS: diskgroup DATAFILE was dismounted
SUCCESS: alter diskgroup DATAFILE dismount force /* ASM SERVER */
SUCCESS: diskgroup TEMPFILE was dismounted
SUCCESS: alter diskgroup TEMPFILE dismount force /* ASM SERVER */
ERROR: PST-initiated MANDATORY DISMOUNT of group TEMPFILE
ERROR: PST-initiated MANDATORY DISMOUNT of group FLASHREC
ERROR: PST-initiated MANDATORY DISMOUNT of group DATAFILE
Wed Mar 14 17:51:07 2012
NOTE: diskgroup resource ora.FLASHREC.dg is offline
NOTE: diskgroup resource ora.DATAFILE.dg is offline
NOTE: diskgroup resource ora.TEMPFILE.dg is offline
Wed Mar 14 17:51:08 2012
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_24250.trc:
ORA-17503: ksfdopn:2 Failed to open file +DATAFILE/<instance name here>/spfile<instance name here>.ora
ORA-15001: diskgroup "DATAFILE" does not exist or is not mounted
Wed Mar 14 17:51:08 2012
SQL> ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
NOTE: cache registered group FLASHREC number=1 incarn=0xfa7fb7ea
SQL> ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
NOTE: cache registered group FLASHREC number=1 incarn=0xfa7fb7ea
NOTE: cache began mount (first) of group FLASHREC number=1 incarn=0xfa7fb7ea
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
WARNING: Read Failed. group:0 disk:1 AU:0 offset:0 size:4096
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
WARNING: Read Failed. group:0 disk:2 AU:0 offset:0 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: cache dismounting (clean) group 1/0xFA7FB7EA (FLASHREC)
NOTE: messaging CKPT to quiesce pins Unix process pid: 27411, image: oracle@dot-oraprd04 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0xFA7FB7EA (FLASHREC)
NOTE: cache ending mount (fail) of group FLASHREC number=1 incarn=0xfa7fb7ea
NOTE: cache deleting context for group FLASHREC 1/0xfa7fb7ea
Wed Mar 14 17:51:08 2012
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 512)
)
GMON dismounting group 1 at 20 for pid 18, osid 27411
ERROR: diskgroup FLASHREC was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "FLASHREC" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "FLASHREC"
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ERROR: ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
ASM Health Checker found 1 new failures

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points