This discussion is archived
9 Replies Latest reply: Dec 17, 2012 6:34 AM by Dude! RSS

Slow IO Performance in Oracle uek 64bit kernel

979966 Newbie
Currently Being Moderated
I am having Slow IO performance issue in Oracle Linux 5.8 uek 64bit kernel for SATA Disk connected via SAN,
and I have tried using dm-multipath and LVM, nep,

and the same SATA disk does not have any issue with RHEL5.8 32bit, with RMAN Backup it is 8-9 hours backup duration,
but in OEL5.8 64bit uek it takes 15Hours to complete RMAN Backup of Oracle R12 DB Backup.



Please help me.
  • 1. Re: Slow IO Performance in Oracle uek 64bit kernel
    Dude! Guru
    Currently Being Moderated
    I suggest to restart the system and select the supplied RHEL 5.8 x86_64 kernel from the Grub boot menu to see if the UEK kernel is the culprit.
  • 2. Re: Slow IO Performance in Oracle uek 64bit kernel
    979966 Newbie
    Currently Being Moderated
    Sorry for confusion,

    I am using Oracle Linux 5.8 64Bit, not RHEL
    and because of some issue we cannot boot in to other one except UEK Kernel.

    and we are facing slow performance issue with SATA Disks / USB Disks in this OEL5.8 64bit OS,
    sometimes it makes server hang and restarts.

    Please help me.
  • 3. Re: Slow IO Performance in Oracle uek 64bit kernel
    Dude! Guru
    Currently Being Moderated
    Can you provide information to be able to reproduce the problem?
  • 4. Re: Slow IO Performance in Oracle uek 64bit kernel
    979966 Newbie
    Currently Being Moderated
    here is iowait for the particular disk

    [root@XXXX ~]# iostat -x 1 5 | grep sdav
    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sdav 34.07 1190.82 104.30 110.41 7861.07 10492.07 85.48 4.07 18.96 2.61 56.05
    sdav 0.00 998.00 0.00 72.00 0.00 10584.00 147.00 2.10 33.56 12.83 92.40
    sdav 0.00 4826.00 1.00 410.00 8.00 39368.00 95.81 3.71 5.19 1.24 50.90
    sdav 0.00 428.00 0.00 36.00 0.00 5872.00 163.11 5.26 168.47 27.78 100.00
    sdav 0.00 3060.00 0.00 333.00 0.00 27424.00 82.35 1.98 7.10 2.06 68.60


    Other OS Details
    [root@XXXXX ~]# uname -a
    Linux awar12db2.aisin-na.com 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:37:40 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

    [root@XXXXX ~]# cat /etc/issue
    Oracle Linux Server release 5.8
    Kernel \r on an \m

    [root@XXXXX ~]# cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 5.8 (Tikanga)


    Sorry, I am not getting any error logs in syslog related to this Slow Performance to provide.

    but, I can explain this IO Slow performance.

    When we use Oracle R11 RMAN Backup on RHEL5.8 32bit OS,
    we don't have any slow performance issue with same SATA Disk,
    Total Backup time with RHEL32 5.8 32 bit OS with this SATA Disk is : 8-9 Hours

    but now with Oracle R12 RMAN Backup on Oracle Enterprise Linux 5.8 64bit OS,
    we are facing this slow performance issuee
    Total Backup time with OEL 5.8 64bit OS with this SATA Disk is : 15 Hours

    Sorry, to ask, please correct me if I wrong
    is this impact of because it is configured with Multipath and LVM.


    Thanks & Regards,

    Prabhakar.D
  • 5. Re: Slow IO Performance in Oracle uek 64bit kernel
    Dude! Guru
    Currently Being Moderated
    Tools like iostat or others measure the workload and can not determine whether or not any running task or configuration is done properly or working as efficently as advertised. Performance tuning of Oracle database, including RMAN is certainly beyond the scope of this forum.

    You can do a simple benchmark to measure the data throughput of your storage device. When you run the tests from different systems or kernels it should give you an idea what to expect. For instance:

    <pre>
    # time dd if=/dev/zero of= */mnt/usb/testfile* bs=1G count=1 oflag=dsync
    # hdparm -tT */dev/sdc2*
    </pre>

    Since the time of your RMAN backup is twice as long, I wonder if you have perhaps another backup process running that you are not aware of.
  • 6. Re: Slow IO Performance in Oracle uek 64bit kernel
    979966 Newbie
    Currently Being Moderated
    Here is output of hdparm test


    hdparm -tT /dev/mapper/VolGroup05-lvol01

    /dev/mapper/VolGroup05-lvol01:
    Timing cached reads: 2752 MB in 2.01 seconds = 1372.14 MB/sec
    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
    Timing buffered disk reads: 244 MB in 3.01 seconds = 81.02 MB/sec
    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

    hdparm -tT /dev/mapper/VolGroup05-lvol01

    /dev/mapper/VolGroup05-lvol01:
    Timing cached reads: 4328 MB in 2.00 seconds = 2167.75 MB/sec
    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
    Timing buffered disk reads: 244 MB in 3.02 seconds = 80.68 MB/sec
    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device


    time dd if=/dev/zero of=/dev/mapper/VolGroup05-lvol01 bs=1G count=1 oflag=dsync
    real 1m18.142s
    user 0m0.000s
    sys 0m0.001s
    1+0 records out
    1073741824 bytes (1.1 GB) copied, 91.1034 seconds, 11.8 MB/s

    PS:This same SATA Disks average output was 120+MB/secs with RHEL432Bit OS.
    but here it (OEL5.8 64Bit) it is less than
  • 7. Re: Slow IO Performance in Oracle uek 64bit kernel
    Dude! Guru
    Currently Being Moderated
    The Oracle UEK x86-64 bit kernel should not slow down your disk access. If it does, then there are unusual compatibility issues or other unusual reasons. Based on your description I have no idea what hardware or I/O interface you are using.

    Performance statistics and statements that disk access is faster in a previous or other version of the Linux kernel is not useful for troubleshooting. I suggest to check the vendor support pages of your disk/adapter subsystem to see if there are any firmware updates or other related information available.
  • 8. Re: Slow IO Performance in Oracle uek 64bit kernel
    979966 Newbie
    Currently Being Moderated
    I request you to suggest me a appropriate Software/Drivers to be used for following HW Configurations, which is recommended & supported for Oracle Linux Officially

    Currently these are Drivers detected by for Server & Storage


    description: System
    product: IBM x3950-[88784RU]-
    vendor: IBM
    serial: 99A4628
    width: 32 bits
    -cpu:0
    description: CPU
    product: Intel(R) Xeon(TM) CPU 3.33GHz
    vendor: Intel Corp.
    physical id: aa
    bus info: cpu@0
    version: Intel Xeon MP Dual-Core
    slot: Node 1 CPU 1
    size: 3340MHz
    capacity: 4GHz
    width: 64 bits
    clock: 133MHz

    *-ide
    description: IDE interface
    product: CSB6 RAID/IDE Controller
    vendor: Broadcom
    physical id: f.1
    bus info: pci@0000:00:0f.1
    logical name: scsi3
    version: a0
    width: 32 bits
    clock: 33MHz
    capabilities: ide bus_master emulated
    configuration: driver=pata_serverworks latency=64 module=pata_serverworks
    *-cdrom
    description: DVD reader
    product: RW/DVD GCC-T10N
    vendor: HL-DT-ST
    physical id: 0.0.0
    bus info: scsi@3:0.0.0
    logical name: /dev/cdrom
    logical name: /dev/cdwriter
    logical name: /dev/dvd
    logical name: /dev/scd0
    logical name: /dev/sr0
    version: 1.00
    capabilities: removable audio cd-r cd-rw dvd
    configuration: ansiversion=5 status=nodisc
    *-storage
    description: RAID bus controller
    product: AAC-RAID
    vendor: Adaptec
    physical id: 2
    bus info: pci@0000:01:02.0
    logical name: scsi0
    version: 02
    width: 64 bits
    clock: 66MHz
    capabilities: storage pm msi pcix bus_master cap_list emulated
    configuration: driver=aacraid latency=240 maxlatency=1 mingnt=1 module=aacraid
    *-disk:0
    description: SCSI Disk
    product: Drive 1
    vendor: IBM
    physical id: 0.0.0
    bus info: scsi@0:0.0.0
    logical name: /dev/sda
    version: V1.0
    serial: C109DCE7
    size: 68GB
    capabilities: partitioned partitioned:dos
    configuration: ansiversion=2
    *-disk:1 UNCLAIMED
    description: SCSI Disk
    product: ST973402SS
    vendor: IBM-ESXS
    physical id: 1.0.0
    bus info: scsi@0:1.0.0
    version: B526
    serial: 3NP1D1FS00009749ZHAG
    capacity: 49GB
    capabilities: 10000rpm
    configuration: ansiversion=5
    *-disk:2 UNCLAIMED
    description: SCSI Disk
    product: ST973401SS
    vendor: IBM-ESXS
    physical id: 1.1.0
    bus info: scsi@0:1.1.0
    version: B51D
    serial: 3LB1MPM6000097365WR7
    capacity: 68GB
    capabilities: 10000rpm
    configuration: ansiversion=5
    *-disk:0
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.2
    bus info: scsi@1:0.0.2
    logical name: /dev/sdb
    version: 0617
    serial: SF64601404
    size: 15GB
    capacity: 15GB
    capabilities: 7200rpm partitioned partitioned:dos
    configuration: ansiversion=5
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.3
    bus info: scsi@1:0.0.3
    logical name: /dev/sdc
    version: 0617
    serial: SF64601404
    size: 150GB
    capacity: 150GB
    capabilities: 7200rpm
    configuration: ansiversion=5
    *-disk:2
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.4
    bus info: scsi@1:0.0.4
    logical name: /dev/sdd
    version: 0617
    serial: SF64601404
    size: 150GB
    capacity: 150GB
    capabilities: 7200rpm
    configuration: ansiversion=5
    *-disk:3
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.5
    bus info: scsi@1:0.0.5
    logical name: /dev/sde
    version: 0617
    serial: SF64601404
    size: 150GB
    capacity: 150GB
    capabilities: 7200rpm
    configuration: ansiversion=5
    *-disk:4
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.7
    -disk:22                                                                 THIS DISK IS HAVING SLOW IO PERFORMANCE PROBLEM*
    description: SCSI Disk
    product: 1726-4xx FAStT
    vendor: IBM
    physical id: 0.0.1d
    bus info: scsi@1:0.0.29
    logical name: /dev/sdx
    version: 0617
    serial: SF64601404
    size: 931GB
    capacity: 931GB
    capabilities: 7200rpm
    configuration: ansiversion=5



    Server
    IBM System X3950
    Internal SAS Drive x 2
    Qlogic 4Gbps HBA Card

    SAN Specs
    IBM DS-3400 SAS
    IBM DS-3400 SATA
    both are connected via HBA Switch and FC Card

    Sorry I missed to mention about
    in this same SAN Storage SAS Disks performance is very well_,
    and we are facing IO Issue only with SATA Disk

    all are configured with dm-multipath and
    SAS Disks are mounted with OCFS2 on two servers
    like RAW --> mpath --> OCFS2

    SATA Disk is mounted with LVM
    RAW --> MPATH --> LVM

    Is it causes because of SATA disk Configured with Multipath & LVM?
    Is it will resolve if we use RDAC Multipath Driver from IBM SDD for DS-3400 for this SATA Disk?

    and also please suggest me a method to get High throughput from this SATA Disks.

    Please help me.
  • 9. Re: Slow IO Performance in Oracle uek 64bit kernel
    Dude! Guru
    Currently Being Moderated
    Please contact Oracle support. Participation in this forum is on a voluntary basis. As such you are not in a position to request any official driver software or a solution from me for your HW configuration. I don't even have to reply to your post if I don't feel like it. If you need help for a production system, getting an Oracle Linux support subscription might be your best option.

    Looking at your problem again, I wonder what records exist that point to an issue with the Oracle Linux kernel or driver. To find out what driver you are using, you can use the "lspci -v" command and filter out the information of the appropriate HBA controller.

    If this is indeed a problem of using a 64-bit or 32-bit driver, I suggest to check the vendor support page (IBM) to see if there is a know issue. Perhaps your SAN or HBA controller or PC BIOS firmware requires a patch for 64-bit compatiblity. And again, to find out if it is a problem with the Oracle UEK kernel, simply boot the system using the RHEL kernel. A problem with the multipath configuration may causes the system to use a drive different than you expect, but afaik multipath does not provide the actual device driver to affect data throughput I/O, nor does LVM.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points