5 Replies Latest reply: Sep 27, 2012 5:22 AM by mohammed farhan RSS

    ASM timeout parameters?

    Dana N
      Oracle: 11.2.0.3 non-RAC (Oracle Restart grid home)
      OS: RHEL Server 5.8

      Can timeout parameters be set for an ASM instance or for any downstream database instances dependent on an ASM instance? Our storage and sysadmins ran a test (failing over a controller). The Oracle database instance detected a problem reaching +FLASHREC on the NetApp device (it was trying to access a control file evidently). Approximately one second later the database instance decided to terminate itself. The ASM instance remained up. But subsequent checking with ASMCD showed no ASM Diskgroup available using an LS command. After bouncing the ASM instance all was well--the diskgroups reappeared and we were able to restart our database instance. A second seems a bit unforgiving. Can any timeout-related parameters be set on the ASM or ASM-client instance to provide more wiggle room during a controller failover?

      ##############################################################################
      Some errors we encountered from the database instance alert log. Further below are errors from the ASM instance's alert log
      ##############################################################################

      Wed Mar 14 04:00:55 2012
      Archived Log entry 89 added for thread 1 sequence 142 ID 0xbb0a69f4 dest 1:
      Wed Mar 14 17:51:06 2012
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 120864
      Additional information: -1
      WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
      Wed Mar 14 17:51:06 2012
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>arc2399.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 120864
      Additional information: -1
      WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
      WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
      WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>arc2399.trc:
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-27061: waiting for async I/Os failed
      Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 16384
      WARNING: Write Failed. group:2 disk:0 AU:59 offset:49152 size:16384
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-15080: synchronous I/O operation to a disk failed
      WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group 2 on disk 0 allocation unit 59
      Wed Mar 14 17:51:06 2012
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 120864
      Additional information: -1
      WARNING: Read Failed. group:2 disk:0 AU:59 offset:16384 size:16384
      WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [2.2142222202] from disk FLASHREC_0000 allocation unit 59 reason error; if possible, will try another mirror side
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-00206: error in writing (block 3, # blocks 1) of control file
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      ORA-15081: failed to submit an I/O operation to a disk
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>ckpt310.trc:
      ORA-00221: error on write to control file
      ORA-00206: error in writing (block 3, # blocks 1) of control file
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      ORA-15081: failed to submit an I/O operation to a disk
      CKPT (ospid: 310): terminating the instance due to error 221
      Errors in file /u01/app/oracle/diag/rdbms/<instance name here>/<instance name here>/trace/<instance name here>m00023737.trc:
      ORA-00204: error in reading (block 1, # blocks 1) of control file
      ORA-00202: control file: '+FLASHREC/<instance name here>/controlfile/current.256.776099703'
      ORA-15081: failed to submit an I/O operation to a disk
      Wed Mar 14 17:51:07 2012
      License high water mark = 8
      Instance terminated by CKPT, pid = 310
      USER (ospid: 24054): terminating the instance
      Instance terminated by USER, pid = 24054


      #######################################
      Some errors we encountered from the ASM instance alert log
      #######################################

      Mon Mar 12 14:56:18 2012
      NOTE: ASMB process exiting due to lack of ASM file activity for 305 seconds
      Wed Mar 14 17:51:06 2012
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 4088
      Additional information: -1
      WARNING: Write Failed. group:1 disk:0 AU:1 offset:1044480 size:4096
      WARNING: Hbeat write to PST disk 0.3916384140 (DATAFILE_0000) in group 1 failed.
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 4088
      Additional information: -1
      WARNING: Write Failed. group:2 disk:0 AU:1 offset:1044480 size:4096
      WARNING: Hbeat write to PST disk 0.3916384141 (FLASHREC_0000) in group 2 failed.
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_gmon_27396.trc:
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 4088
      Additional information: -1
      WARNING: Write Failed. group:3 disk:0 AU:1 offset:1044480 size:4096
      WARNING: Hbeat write to PST disk 0.3916384142 (TEMPFILE_0000) in group 3 failed.
      Wed Mar 14 17:51:06 2012
      NOTE: process b000+asm (23739) initiating offline of disk 0.3916384140 (DATAFILE_0000) with mask 0x7e in group 1
      WARNING: Disk 0 (DATAFILE_0000) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
      NOTE: initiating PST update: grp = 1, dsk = 0/0xe96f478c, mask = 0x6a, op = clear
      Wed Mar 14 17:51:06 2012
      NOTE: process b001+asm (23753) initiating offline of disk 0.3916384141 (FLASHREC_0000) with mask 0x7e in group 2
      WARNING: Disk 0 (FLASHREC_0000) in group 2 in mode 0x7f is now being taken offline on ASM inst 1
      NOTE: initiating PST update: grp = 2, dsk = 0/0xe96f478d, mask = 0x6a, op = clear
      GMON updating disk modes for group 1 at 13 for pid 20, osid 23739
      ERROR: no read quorum in group: required 1, found 0 disks
      Wed Mar 14 17:51:06 2012
      NOTE: process b002+asm (23791) initiating offline of disk 0.3916384142 (TEMPFILE_0000) with mask 0x7e in group 3
      WARNING: Disk 0 (TEMPFILE_0000) in group 3 in mode 0x7f is now being taken offline on ASM inst 1
      NOTE: initiating PST update: grp = 3, dsk = 0/0xe96f478e, mask = 0x6a, op = clear
      GMON updating disk modes for group 2 at 14 for pid 23, osid 23753
      ERROR: no read quorum in group: required 1, found 0 disks
      Wed Mar 14 17:51:06 2012
      NOTE: cache dismounting (not clean) group 1/0x7FAFB779 (DATAFILE)
      NOTE: messaging CKPT to quiesce pins Unix process pid: 23826, image: oracle@dot-oraprd04 (B003)
      Wed Mar 14 17:51:06 2012
      NOTE: halting all I/Os to diskgroup 1 (DATAFILE)
      Wed Mar 14 17:51:06 2012
      NOTE: LGWR doing non-clean dismount of group 1 (DATAFILE)
      NOTE: LGWR sync ABA=6.6196 last written ABA 6.6196
      NOTE: cache dismounted group 1/0x7FAFB779 (DATAFILE)
      SQL> alter diskgroup DATAFILE dismount force /* ASM SERVER */
      Wed Mar 14 17:51:06 2012
      NOTE: cache dismounting (not clean) group 2/0x7FAFB77A (FLASHREC)
      NOTE: messaging CKPT to quiesce pins Unix process pid: 23836, image: oracle@dot-oraprd04 (B004)
      NOTE: halting all I/Os to diskgroup 2 (FLASHREC)
      NOTE: LGWR doing non-clean dismount of group 2 (FLASHREC)
      NOTE: LGWR sync ABA=5.1120 last written ABA 5.1120
      GMON updating disk modes for group 3 at 15 for pid 25, osid 23791
      ERROR: no read quorum in group: required 1, found 0 disks
      NOTE: cache dismounted group 2/0x7FAFB77A (FLASHREC)
      SQL> alter diskgroup FLASHREC dismount force /* ASM SERVER */
      Wed Mar 14 17:51:06 2012
      NOTE: cache dismounting (not clean) group 3/0x7FAFB77B (TEMPFILE)
      NOTE: messaging CKPT to quiesce pins Unix process pid: 23838, image: oracle@dot-oraprd04 (B005)
      NOTE: halting all I/Os to diskgroup 3 (TEMPFILE)
      NOTE: LGWR doing non-clean dismount of group 3 (TEMPFILE)
      NOTE: LGWR sync ABA=6.11 last written ABA 6.11
      NOTE: cache dismounted group 3/0x7FAFB77B (TEMPFILE)
      SQL> alter diskgroup TEMPFILE dismount force /* ASM SERVER */
      WARNING: Offline of disk 0 (TEMPFILE_0000) in group 3 and mode 0x7f failed on ASM inst 1
      WARNING: Offline of disk 0 (DATAFILE_0000) in group 1 and mode 0x7f failed on ASM inst 1
      WARNING: Offline of disk 0 (FLASHREC_0000) in group 2 and mode 0x7f failed on ASM inst 1
      Wed Mar 14 17:51:07 2012
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      Wed Mar 14 17:51:07 2012
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      Wed Mar 14 17:51:07 2012
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ASM Health Checker found 1 new failures
      ASM Health Checker found 1 new failures
      ASM Health Checker found 1 new failures
      Wed Mar 14 17:51:07 2012
      NOTE: ASM client <instance name here>:<instance name here> disconnected unexpectedly.
      NOTE: check client alert log.
      NOTE: Trace records dumped in trace file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_322.trc
      Wed Mar 14 17:51:07 2012
      NOTE: cache deleting context for group FLASHREC 2/0x7fafb77a
      NOTE: cache deleting context for group TEMPFILE 3/0x7fafb77b
      NOTE: cache deleting context for group DATAFILE 1/0x7fafb779
      GMON dismounting group 2 at 16 for pid 27, osid 23836
      GMON dismounting group 1 at 17 for pid 26, osid 23826
      NOTE: Disk in mode 0x8 marked for de-assignment
      GMON dismounting group 3 at 18 for pid 28, osid 23838
      NOTE: Disk in mode 0x8 marked for de-assignment
      NOTE: Disk in mode 0x8 marked for de-assignment
      SUCCESS: diskgroup FLASHREC was dismounted
      SUCCESS: alter diskgroup FLASHREC dismount force /* ASM SERVER */
      SUCCESS: diskgroup DATAFILE was dismounted
      SUCCESS: alter diskgroup DATAFILE dismount force /* ASM SERVER */
      SUCCESS: diskgroup TEMPFILE was dismounted
      SUCCESS: alter diskgroup TEMPFILE dismount force /* ASM SERVER */
      ERROR: PST-initiated MANDATORY DISMOUNT of group TEMPFILE
      ERROR: PST-initiated MANDATORY DISMOUNT of group FLASHREC
      ERROR: PST-initiated MANDATORY DISMOUNT of group DATAFILE
      Wed Mar 14 17:51:07 2012
      NOTE: diskgroup resource ora.FLASHREC.dg is offline
      NOTE: diskgroup resource ora.DATAFILE.dg is offline
      NOTE: diskgroup resource ora.TEMPFILE.dg is offline
      Wed Mar 14 17:51:08 2012
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_24250.trc:
      ORA-17503: ksfdopn:2 Failed to open file +DATAFILE/<instance name here>/spfile<instance name here>.ora
      ORA-15001: diskgroup "DATAFILE" does not exist or is not mounted
      Wed Mar 14 17:51:08 2012
      SQL> ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
      NOTE: cache registered group FLASHREC number=1 incarn=0xfa7fb7ea
      SQL> ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
      NOTE: cache registered group FLASHREC number=1 incarn=0xfa7fb7ea
      NOTE: cache began mount (first) of group FLASHREC number=1 incarn=0xfa7fb7ea
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
      ORA-27061: waiting for async I/Os failed
      Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 4096
      WARNING: Read Failed. group:0 disk:1 AU:0 offset:0 size:4096
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
      ORA-27061: waiting for async I/Os failed
      Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 4096
      WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
      Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27411.trc:
      ORA-27061: waiting for async I/Os failed
      Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 4096
      WARNING: Read Failed. group:0 disk:2 AU:0 offset:0 size:4096
      ERROR: no read quorum in group: required 2, found 0 disks
      NOTE: cache dismounting (clean) group 1/0xFA7FB7EA (FLASHREC)
      NOTE: messaging CKPT to quiesce pins Unix process pid: 27411, image: oracle@dot-oraprd04 (TNS V1-V3)
      NOTE: dbwr not being msg'd to dismount
      NOTE: lgwr not being msg'd to dismount
      NOTE: cache dismounted group 1/0xFA7FB7EA (FLASHREC)
      NOTE: cache ending mount (fail) of group FLASHREC number=1 incarn=0xfa7fb7ea
      NOTE: cache deleting context for group FLASHREC 1/0xfa7fb7ea
      Wed Mar 14 17:51:08 2012
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      ERROR: -9(Error 27061, OS Error (Linux-x86_64 Error: 5: Input/output error
      Additional information: -1
      Additional information: 512)
      )
      GMON dismounting group 1 at 20 for pid 18, osid 27411
      ERROR: diskgroup FLASHREC was not mounted
      ORA-15032: not all alterations performed
      ORA-15017: diskgroup "FLASHREC" cannot be mounted
      ORA-15063: ASM discovered an insufficient number of disks for diskgroup "FLASHREC"
      ORA-15080: synchronous I/O operation to a disk failed
      ORA-15080: synchronous I/O operation to a disk failed
      ORA-15080: synchronous I/O operation to a disk failed
      ERROR: ALTER DISKGROUP FLASHREC MOUNT /* asm agent *//* {0:5:72} */
      ASM Health Checker found 1 new failures