As far as I know, iostat only measures physical block I/O operations and does not read the page cache, storage controller cache, or the Oracle database buffer cache. I think it cannot be used to measure ASM I/O performance and sizes.
Anyway, :) iostat is very well aligned with storage controller statistic. My storage shows same result like iostat. It means that "someone" is issuing these 32K IOs to storage. My question is who is splitting Oracle 1M IO into chunks of 32K IOs? Question is very serious, as it is a significant penalty on IO throughput. We can increase backup speed in 4 times if Oracle process would be able to read using 512K requests and not 32K.
I would proceed with SR on that.
Just speculating here, but are you using the large pool with rman? On an older version on a different OS, I think I noticed that using the large pool made for for much smoother and larger I/O (though I could be misremembering, as the problem I was solving at that time was OS-specific I/O buffer fragmentation).
Anyways, it sounds like you have a stripe width v. AU conflict. Does the ASMCMD volinfo command tell us anything?
Dude, we are on Xen HVM +PV guest. Physycal storage is IBM V7000 connected through 4 8Gb FC ports. I can not pos graphs here,but what is interesting:
When we are doing storage calibration with Orion tool, it shows performance close to what we expect: 1.8GB/sec.
We use following Orion command line and lun file:
./orion_linux_x86-64 -run advanced -testname sas_20121206 -matrix point -num_small 0 -num_large 1 -size_large 512 -num_disks 35 -type seq -num_streamIO 10 - simulate raid0 -cache_size 0 -duration 120 -verbose
ORION: ORacle IO Numbers -- Version 126.96.36.199.0
Test will take approximately 3 minutes
Larger caches may take longer
Name: /dev/xvdb1 Size: 1198396539392
Name: /dev/xvde1 Size: 1198396539392
Name: /dev/xvdf1 Size: 1198396539392
Name: /dev/xvdg1 Size: 1198396539392
Name: /dev/xvdj1 Size: 1198396539392
Name: /dev/xvdk1 Size: 1198396539392
Name: /dev/xvdl1 Size: 1198396539392
Name: /dev/xvdb2 Size: 1198396539392
Name: /dev/xvde2 Size: 1198396539392
Name: /dev/xvdf2 Size: 1198396539392
Name: /dev/xvdg2 Size: 1198396539392
Name: /dev/xvdj2 Size: 1198396539392
Name: /dev/xvdk2 Size: 1198396539392
Name: /dev/xvdl2 Size: 1198396539392
14 FILEs found.
In fact when we are looking to storage statistics or IOSTAT during these tests (they perfectly match) we see that storage is receivng ~40000 requests over FC apron 6000 per LUN this gives ~40K per request. It is confusing for me, because we suppose to issue 512K reads
With DB process picture is a bit different, but still confusing.
While searching on the web, I found interesting report (i am not sure I could post links here, anyway):
If you look through pages 8-9 you will find:
This iostat output shows that, although Oracle was making 1MB read calls, the Linux kernel was breaking these calls into multiple smaller calls – in this case, around 350KB in size. This is a function of the Linux block layer, which was operating correctly based on the default configuration of the system. No tuning of the kernel-block layer took place for this testing because, although more throughput could be obtained by eliminating this request fracturing, as long as this phenomenon was the same for both HBA types then the comparison remained valid. Indeed, it reflected the reality that many end-user environments do not tune the block layer at all.
Seems that James has some information to share :)
You mean on VM? or on host?
On VM in fact we don't have such file at all.
[root@lnxru0240vg081 ~]# cat /etc/modprobe.conf
cat: /etc/modprobe.conf: No such file or directory
On VM host - nothing special:
[root@ovsru0240vh021 ~]# cat /etc/modprobe.conf
alias scsi_hostadapter shpchp
#alias eth0 bnx2
#alias eth1 bnx2
alias eth2 be2net
alias eth3 be2net
alias scsi_hostadapter1 megaraid_sas
alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter3 lpfc
alias scsi_hostadapter4 usb-storage
alias eth0 bnx2
alias eth1 bnx2
install lpfc /sbin/modprobe bnx2 ; /sbin/modprobe -i lpfc ; true
Can you provide more information, e.g. kernel (uname -r) of Dom0 and DomU machine?
If the virtual machine is on Oracle Linux 6, you should have a directory named /etc/modprobe.d/. The reason I'm asking is because storage drivers, e.g. lpfc have settings that can be customized to affect channel count and queue sizes and thereby I/O size.
Also I think this post should probably be moved to the Oracle VM forum for closer topic alignment.