2 Replies Latest reply on Oct 3, 2012 9:17 AM by Phani6039

    getting I/O error and not sure why?

    Phani6039
      Hi,

      Recently we are getting I/O error messages in our message file which could be pushing our physical server into hung state.
      The physical machine is running oracle vm server 2.2 which hosts 3 other VMs.
      df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/cciss/c0d0p2     9.7G  1.2G  8.1G  12% / 
      /dev/cciss/c0d0p1      99M   57M   38M  61% /boot
      tmpfs                 288M     0  288M   0% /dev/shm
      /dev/mapper/oracle1p1
                            269G  187G   83G  70% /var/ovs/mount/3114099BE91240B98001AACDF0881F6F
      /dev/mapper/oracle2p1
                             69G   49G   20G  71% /var/ovs/mount/0E96744970BB4A3B9D06780F80BE593A
      /dev/mapper/oracle3   100G   33G   68G  33% /var/ovs/mount/0456DAB2C1B546B09E1F244DF3D73334
       
      error message: Got huge list but posted only some.
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdba, sector 209715072
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:17: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdba, sector 0
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:24: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbb, sector 10485632
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:24: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbb, sector 0
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:25: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbc, sector 10485632
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:25: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbc, sector 0
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:26: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbd, sector 10485632
      Oct  1 15:53:16 oracle 2 kernel: sd 3:0:1:26: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:16 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:16 oracle 2 kernel: 
      Oct  1 15:53:16 oracle 2 kernel: end_request: I/O error, dev sdbd, sector 0
      Oct  1 15:53:17 oracle 2 kernel: sd 3:0:1:30: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:17 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:17 oracle 2 kernel: 
      Oct  1 15:53:17 oracle 2 kernel: end_request: I/O error, dev sdbf, sector 562814464
      Oct  1 15:53:17 oracle 2 kernel: sd 3:0:1:30: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:17 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:17 oracle 2 kernel: 
      Oct  1 15:53:17 oracle 2 kernel: end_request: I/O error, dev sdbf, sector 0
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:36: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbl, sector 2251258240
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:36: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbl, sector 0
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:40: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbn, sector 209715072
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:40: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbn, sector 0
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:41: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbo, sector 209715072
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:41: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbo, sector 0
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:42: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbp, sector 209715072
      Oct  1 15:53:18 oracle 2 kernel: sd 3:0:1:42: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:18 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:18 oracle 2 kernel: 
      Oct  1 15:53:18 oracle 2 kernel: end_request: I/O error, dev sdbp, sector 0
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:43: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbq, sector 209715072
      Oct  1 15:53:19 oracle 2 kernel: printk: 32 messages suppressed.
      Oct  1 15:53:19 oracle 2 kernel: Buffer I/O error on device sdbq, logical block 26214384
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:43: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbq, sector 0
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:44: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbr, sector 209715072
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:44: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbr, sector 0
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:45: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbs, sector 209715072
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:45: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbs, sector 0
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:46: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbt, sector 209715072
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:46: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbt, sector 0
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:50: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbu, sector 209715072
      Oct  1 15:53:19 oracle 2 kernel: sd 3:0:1:50: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:19 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:19 oracle 2 kernel: 
      Oct  1 15:53:19 oracle 2 kernel: end_request: I/O error, dev sdbu, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:51: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbv, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:51: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbv, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:52: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbw, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:52: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbw, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:53: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbx, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:53: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbx, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:54: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdby, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:54: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdby, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:55: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbz, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:55: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdbz, sector 0
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:56: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdca, sector 209715072
      Oct  1 15:53:20 oracle 2 kernel: sd 3:0:1:56: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:20 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:20 oracle 2 kernel: 
      Oct  1 15:53:20 oracle 2 kernel: end_request: I/O error, dev sdca, sector 0
      Oct  1 15:53:21 oracle 2 kernel: sd 3:0:1:57: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:21 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:21 oracle 2 kernel: 
      Oct  1 15:53:21 oracle 2 kernel: end_request: I/O error, dev sdcb, sector 209715072
      Oct  1 15:53:21 oracle 2 kernel: sd 3:0:1:57: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:21 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:21 oracle 2 kernel: 
      Oct  1 15:53:21 oracle 2 kernel: end_request: I/O error, dev sdcb, sector 0
      Oct  1 15:53:21 oracle 2 kernel: sd 3:0:1:58: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:21 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:21 oracle 2 kernel: 
      Oct  1 15:53:21 oracle 2 kernel: end_request: I/O error, dev sdcc, sector 209715072
      Oct  1 15:53:21 oracle 2 kernel: sd 3:0:1:58: Device not ready: <6>: Current: sense key: Not Ready
      Oct  1 15:53:21 oracle 2 kernel:     Add. Sense: Logical unit not ready, manual intervention required
      Oct  1 15:53:21 oracle 2 kernel: 
      Oct  1 15:53:21 oracle 2 kernel: end_request: I/O error, dev sdcc, sector 0
       
      Other outputs for some of the commands are as below
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">mounted.ocfs2 -d</div>
      Device                FS     UUID                                  Label
      /dev/cciss/c0d0p3     ocfs2  053ace1d-23e5-4fd3-a88f-793595ed3973  
      /dev/sdl1             ocfs2  cf18a0af-ece8-4d11-b397-aef3ea726c61  /u02
      /dev/sdq              ocfs2  a2758f58-3997-4393-b295-52afff1d15e5  
      /dev/sdq1             ocfs2  01697c82-bd1a-4f12-be14-fbda7b9418d5  
      /dev/sdbe             ocfs2  102e738b-f458-4986-8e31-061eda41c04c  100GB
      /dev/sdbg             ocfs2  06a20911-e123-4020-b1a6-9010f6d1315c  
      /dev/sdbg1            ocfs2  8842ce25-231b-4c54-b6c9-fe5d44170209  
      /dev/dm-29            ocfs2  102e738b-f458-4986-8e31-061eda41c04c  100GB
      /dev/dm-30            ocfs2  a2758f58-3997-4393-b295-52afff1d15e5  
      /dev/dm-31            ocfs2  06a20911-e123-4020-b1a6-9010f6d1315c  
      /dev/dm-42            ocfs2  8842ce25-231b-4c54-b6c9-fe5d44170209  
      /dev/dm-46            ocfs2  01697c82-bd1a-4f12-be14-fbda7b9418d5  
      /dev/dm-50            ocfs2  cf18a0af-ece8-4d11-b397-aef3ea726c61  /u02
       
       [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">service multipathd status</div>
      multipathd (pid 10369) is running...
       
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">ll /sys/class/fc_host/</div>
      total 0
      drwxr-xr-x 3 root root 0 Oct  1 09:22 host2
      drwxr-xr-x 3 root root 0 Oct  1 09:22 host3
       
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">lspci |grep -i fibre</div>
      0a:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
      0d:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
       
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">vgdisplay -v</div>
          Finding all volume groups
        /dev/sda: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdcc: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdr: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbn: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdcd: read failed after 0 of 4096 at 0: Input/output error
        /dev/sds: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbo: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdt: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdaz: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbp: read failed after 0 of 4096 at 0: Input/output error
        /dev/sde: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdu: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdba: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbq: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdf: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdv: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbb: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbr: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdg: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbc: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbs: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdh: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdx: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbd: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbt: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbu: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdj: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdap: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbf: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbv: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbw: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdar: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbx: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdby: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbz: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdca: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdp: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdbl: read failed after 0 of 4096 at 0: Input/output error
        /dev/sdcb: read failed after 0 of 4096 at 0: Input/output error
       
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">ll /dev/mapper/</div>
      total 0
      crw------- 1 root root  10, 61 Oct  1 09:22 control
      brw-rw---- 1 root disk 253, 36 Oct  1 09:22 data1
      brw-rw---- 1 root disk 253, 74 Oct  1 09:22 data1p1
      brw-rw---- 1 root disk 253, 25 Oct  1 09:22 diaglog1
      brw-rw---- 1 root disk 253, 50 Oct  1 09:22 diaglog1p1
      brw-rw---- 1 root disk 253, 35 Oct  1 09:22 fra1
      brw-rw---- 1 root disk 253, 63 Oct  1 09:22 fra1p1
      brw-rw---- 1 root disk 253, 26 Oct  1 09:22 L24L1
      brw-rw---- 1 root disk 253, 72 Oct  1 09:22 L24L1p1
      brw-rw---- 1 root disk 253, 27 Oct  1 09:22 L25L2
      brw-rw---- 1 root disk 253, 70 Oct  1 09:22 L25L2p1
      brw-rw---- 1 root disk 253, 28 Oct  1 09:22 L26L3
      brw-rw---- 1 root disk 253, 68 Oct  1 09:22 L26L3p1
      brw-rw---- 1 root disk 253, 38 Oct  1 09:22 L40F1
      brw-rw---- 1 root disk 253, 71 Oct  1 09:22 L40F1p1
      brw-rw---- 1 root disk 253, 39 Oct  1 09:22 L41F2
      brw-rw---- 1 root disk 253, 69 Oct  1 09:22 L41F2p1
      brw-rw---- 1 root disk 253,  0 Oct  1 09:22 L42F3
      brw-rw---- 1 root disk 253, 51 Oct  1 09:22 L42F3p1
      brw-rw---- 1 root disk 253,  1 Oct  1 09:22 L43F4
      brw-rw---- 1 root disk 253, 54 Oct  1 09:22 L43F4p1
      brw-rw---- 1 root disk 253,  2 Oct  1 09:22 L44F5
      brw-rw---- 1 root disk 253, 57 Oct  1 09:22 L44F5p1
      brw-rw---- 1 root disk 253,  3 Oct  1 09:22 L45F6
      brw-rw---- 1 root disk 253, 59 Oct  1 09:22 L45F6p1
      brw-rw---- 1 root disk 253,  4 Oct  1 09:22 L46F7
      brw-rw---- 1 root disk 253, 58 Oct  1 09:22 L46F7p1
      brw-rw---- 1 root disk 253,  5 Oct  1 09:22 L50D1
      brw-rw---- 1 root disk 253, 62 Oct  1 09:22 L50D1p1
      brw-rw---- 1 root disk 253,  6 Oct  1 09:22 L51D2
      brw-rw---- 1 root disk 253, 65 Oct  1 09:22 L51D2p1
      brw-rw---- 1 root disk 253,  7 Oct  1 09:22 L52D3
      brw-rw---- 1 root disk 253, 67 Oct  1 09:22 L52D3p1
      brw-rw---- 1 root disk 253,  8 Oct  1 09:22 L53D4
      brw-rw---- 1 root disk 253, 66 Oct  1 09:22 L53D4p1
      brw-rw---- 1 root disk 253,  9 Oct  1 09:22 L54D5
      brw-rw---- 1 root disk 253, 64 Oct  1 09:22 L54D5p1
      brw-rw---- 1 root disk 253, 10 Oct  1 09:22 L55D6
      brw-rw---- 1 root disk 253, 61 Oct  1 09:22 L55D6p1
      brw-rw---- 1 root disk 253, 11 Oct  1 09:22 L56D7
      brw-rw---- 1 root disk 253, 56 Oct  1 09:22 L56D7p1
      brw-rw---- 1 root disk 253, 12 Oct  1 09:22 L57D8
      brw-rw---- 1 root disk 253, 55 Oct  1 09:22 L57D8p1
      brw-rw---- 1 root disk 253, 13 Oct  1 09:22 L58D9
      brw-rw---- 1 root disk 253, 52 Oct  1 09:22 L58D9p1
      brw-rw---- 1 root disk 253, 14 Oct  1 09:22 L59D10
      brw-rw---- 1 root disk 253, 40 Oct  1 09:22 L59D10p1
      brw-rw---- 1 root disk 253, 32 Oct  1 09:22 logs1
      brw-rw---- 1 root disk 253, 60 Oct  1 09:22 logs1p1
      brw-rw---- 1 root disk 253, 33 Oct  1 09:22 logs2
      brw-rw---- 1 root disk 253, 53 Oct  1 09:22 logs2p1
      brw-rw---- 1 root disk 253, 34 Oct  1 09:22 logs3
      brw-rw---- 1 root disk 253, 49 Oct  1 09:22 logs3p1
      brw-rw---- 1 root disk 253, 15 Oct  1 09:22 mpath42
      brw-rw---- 1 root disk 253, 16 Oct  1 09:22 mpath43
      brw-rw---- 1 root disk 253, 17 Oct  1 09:22 mpath44
      brw-rw---- 1 root disk 253, 18 Oct  1 09:22 mpath68
      brw-rw---- 1 root disk 253, 47 Oct  1 09:22 mpath68p1
      brw-rw---- 1 root disk 253, 19 Oct  1 09:22 mpath69
      brw-rw---- 1 root disk 253, 43 Oct  1 09:22 mpath69p1
      brw-rw---- 1 root disk 253, 20 Oct  1 09:22 mpath70
      brw-rw---- 1 root disk 253, 48 Oct  1 09:22 mpath70p1
      brw-rw---- 1 root disk 253, 21 Oct  1 09:22 mpath71
      brw-rw---- 1 root disk 253, 44 Oct  1 09:22 mpath71p1
      brw-rw---- 1 root disk 253, 22 Oct  1 09:22 mpath72
      brw-rw---- 1 root disk 253, 41 Oct  1 09:22 mpath72p1
      brw-rw---- 1 root disk 253, 23 Oct  1 09:22 mpath73
      brw-rw---- 1 root disk 253, 75 Oct  1 09:22 mpath73p1
      brw-rw---- 1 root disk 253, 24 Oct  1 09:22 mpath74
      brw-rw---- 1 root disk 253, 73 Oct  1 09:22 mpath74p1
      brw-rw---- 1 root disk 253, 30 Oct  1 09:22 oracle1
      brw-rw---- 1 root disk 253, 46 Oct  1 09:22 oracle1p1
      brw-rw---- 1 root disk 253, 31 Oct  1 09:22 oracle2
      brw-rw---- 1 root disk 253, 42 Oct  1 09:22 oracle2p1
      brw-rw---- 1 root disk 253, 29 Oct  1 09:22 oracle3
      brw-rw---- 1 root disk 253, 37 Oct  1 09:22 test1
      brw-rw---- 1 root disk 253, 45 Oct  1 09:22 test1p1
       
      [root@oracle 2]:/var/ovs/mount
      <div class="jive-quote">dmsetup ls</div>
      mpath70p1       (253, 48)
      L59D10  (253, 14)
      L45F6p1 (253, 59)
      mpath71p1       (253, 44)
      mpath72 (253, 22)
      oracle2 (253, 31)
      L57D8p1 (253, 55)
      mpath72p1       (253, 41)
      mpath69 (253, 19)
      mpath71 (253, 21)
      mpath73p1       (253, 75)
      test1   (253, 37)
      oracle1 (253, 30)
      L25L2p1 (253, 70)
      L41F2p1 (253, 69)
      mpath74p1       (253, 73)
      mpath68 (253, 18)
      L53D4p1 (253, 66)
      mpath70 (253, 20)
      L46F7p1 (253, 58)
      L58D9p1 (253, 52)
      diaglog1p1      (253, 50)
      L59D10p1        (253, 40)
      logs3   (253, 34)
      L26L3p1 (253, 68)
      L42F3p1 (253, 51)
      diaglog1        (253, 25)
      L58D9   (253, 13)
      logs2   (253, 33)
      L57D8   (253, 12)
      L54D5p1 (253, 64)
      fra1p1  (253, 63)
      oracle1p1       (253, 46)
      test1p1 (253, 45)
      L56D7   (253, 11)
      L46F7   (253, 4)
      logs1   (253, 32)
      L55D6   (253, 10)
      oracle2p1       (253, 42)
      L45F6   (253, 3)
      fra1    (253, 35)
      L54D5   (253, 9)
      L44F5   (253, 2)
      L53D4   (253, 8)
      L50D1p1 (253, 62)
      L43F4   (253, 1)
      L52D3   (253, 7)
      L26L3   (253, 28)
      L42F3   (253, 0)
      L43F4p1 (253, 54)
      L51D2   (253, 6)
      mpath68p1       (253, 47)
      L41F2   (253, 39)
      L25L2   (253, 27)
      L50D1   (253, 5)
      L55D6p1 (253, 61)
      logs1p1 (253, 60)
      L40F1   (253, 38)
      L24L1   (253, 26)
      mpath69p1       (253, 43)
      logs2p1 (253, 53)
      mpath44 (253, 17)
      logs3p1 (253, 49)
      data1p1 (253, 74)
      L51D2p1 (253, 65)
      mpath43 (253, 16)
      L44F5p1 (253, 57)
      L56D7p1 (253, 56)
      mpath42 (253, 15)
      mpath74 (253, 24)
      L24L1p1 (253, 72)
      L40F1p1 (253, 71)
      data1   (253, 36)
      mpath73 (253, 23)
      L52D3p1 (253, 67)
      oracle3 (253, 29)
       
       
      Able to see got 2 hba and not sure where to start from. Any help regarding them will be great. Think could be issue from storage end with zonnig or something which i am not sure of.

      Thanks,
      Phani