9 Replies Latest reply: Dec 17, 2012 8:34 AM by Catch~22 RSS

    Slow IO Performance in Oracle uek 64bit kernel

    979966
      I am having Slow IO performance issue in Oracle Linux 5.8 uek 64bit kernel for SATA Disk connected via SAN,
      and I have tried using dm-multipath and LVM, nep,

      and the same SATA disk does not have any issue with RHEL5.8 32bit, with RMAN Backup it is 8-9 hours backup duration,
      but in OEL5.8 64bit uek it takes 15Hours to complete RMAN Backup of Oracle R12 DB Backup.



      Please help me.
        • 1. Re: Slow IO Performance in Oracle uek 64bit kernel
          Catch~22
          I suggest to restart the system and select the supplied RHEL 5.8 x86_64 kernel from the Grub boot menu to see if the UEK kernel is the culprit.
          • 2. Re: Slow IO Performance in Oracle uek 64bit kernel
            979966
            Sorry for confusion,

            I am using Oracle Linux 5.8 64Bit, not RHEL
            and because of some issue we cannot boot in to other one except UEK Kernel.

            and we are facing slow performance issue with SATA Disks / USB Disks in this OEL5.8 64bit OS,
            sometimes it makes server hang and restarts.

            Please help me.
            • 3. Re: Slow IO Performance in Oracle uek 64bit kernel
              Catch~22
              Can you provide information to be able to reproduce the problem?
              • 4. Re: Slow IO Performance in Oracle uek 64bit kernel
                979966
                here is iowait for the particular disk

                [root@XXXX ~]# iostat -x 1 5 | grep sdav
                Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
                sdav 34.07 1190.82 104.30 110.41 7861.07 10492.07 85.48 4.07 18.96 2.61 56.05
                sdav 0.00 998.00 0.00 72.00 0.00 10584.00 147.00 2.10 33.56 12.83 92.40
                sdav 0.00 4826.00 1.00 410.00 8.00 39368.00 95.81 3.71 5.19 1.24 50.90
                sdav 0.00 428.00 0.00 36.00 0.00 5872.00 163.11 5.26 168.47 27.78 100.00
                sdav 0.00 3060.00 0.00 333.00 0.00 27424.00 82.35 1.98 7.10 2.06 68.60


                Other OS Details
                [root@XXXXX ~]# uname -a
                Linux awar12db2.aisin-na.com 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:37:40 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

                [root@XXXXX ~]# cat /etc/issue
                Oracle Linux Server release 5.8
                Kernel \r on an \m

                [root@XXXXX ~]# cat /etc/redhat-release
                Red Hat Enterprise Linux Server release 5.8 (Tikanga)


                Sorry, I am not getting any error logs in syslog related to this Slow Performance to provide.

                but, I can explain this IO Slow performance.

                When we use Oracle R11 RMAN Backup on RHEL5.8 32bit OS,
                we don't have any slow performance issue with same SATA Disk,
                Total Backup time with RHEL32 5.8 32 bit OS with this SATA Disk is : 8-9 Hours

                but now with Oracle R12 RMAN Backup on Oracle Enterprise Linux 5.8 64bit OS,
                we are facing this slow performance issuee
                Total Backup time with OEL 5.8 64bit OS with this SATA Disk is : 15 Hours

                Sorry, to ask, please correct me if I wrong
                is this impact of because it is configured with Multipath and LVM.


                Thanks & Regards,

                Prabhakar.D
                • 5. Re: Slow IO Performance in Oracle uek 64bit kernel
                  Catch~22
                  Tools like iostat or others measure the workload and can not determine whether or not any running task or configuration is done properly or working as efficently as advertised. Performance tuning of Oracle database, including RMAN is certainly beyond the scope of this forum.

                  You can do a simple benchmark to measure the data throughput of your storage device. When you run the tests from different systems or kernels it should give you an idea what to expect. For instance:

                  <pre>
                  # time dd if=/dev/zero of= */mnt/usb/testfile* bs=1G count=1 oflag=dsync
                  # hdparm -tT */dev/sdc2*
                  </pre>

                  Since the time of your RMAN backup is twice as long, I wonder if you have perhaps another backup process running that you are not aware of.
                  • 6. Re: Slow IO Performance in Oracle uek 64bit kernel
                    979966
                    Here is output of hdparm test


                    hdparm -tT /dev/mapper/VolGroup05-lvol01

                    /dev/mapper/VolGroup05-lvol01:
                    Timing cached reads: 2752 MB in 2.01 seconds = 1372.14 MB/sec
                    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
                    Timing buffered disk reads: 244 MB in 3.01 seconds = 81.02 MB/sec
                    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

                    hdparm -tT /dev/mapper/VolGroup05-lvol01

                    /dev/mapper/VolGroup05-lvol01:
                    Timing cached reads: 4328 MB in 2.00 seconds = 2167.75 MB/sec
                    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
                    Timing buffered disk reads: 244 MB in 3.02 seconds = 80.68 MB/sec
                    HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device


                    time dd if=/dev/zero of=/dev/mapper/VolGroup05-lvol01 bs=1G count=1 oflag=dsync
                    real 1m18.142s
                    user 0m0.000s
                    sys 0m0.001s
                    1+0 records out
                    1073741824 bytes (1.1 GB) copied, 91.1034 seconds, 11.8 MB/s

                    PS:This same SATA Disks average output was 120+MB/secs with RHEL432Bit OS.
                    but here it (OEL5.8 64Bit) it is less than
                    • 7. Re: Slow IO Performance in Oracle uek 64bit kernel
                      Catch~22
                      The Oracle UEK x86-64 bit kernel should not slow down your disk access. If it does, then there are unusual compatibility issues or other unusual reasons. Based on your description I have no idea what hardware or I/O interface you are using.

                      Performance statistics and statements that disk access is faster in a previous or other version of the Linux kernel is not useful for troubleshooting. I suggest to check the vendor support pages of your disk/adapter subsystem to see if there are any firmware updates or other related information available.
                      • 8. Re: Slow IO Performance in Oracle uek 64bit kernel
                        979966
                        I request you to suggest me a appropriate Software/Drivers to be used for following HW Configurations, which is recommended & supported for Oracle Linux Officially

                        Currently these are Drivers detected by for Server & Storage


                        description: System
                        product: IBM x3950-[88784RU]-
                        vendor: IBM
                        serial: 99A4628
                        width: 32 bits
                        -cpu:0
                        description: CPU
                        product: Intel(R) Xeon(TM) CPU 3.33GHz
                        vendor: Intel Corp.
                        physical id: aa
                        bus info: cpu@0
                        version: Intel Xeon MP Dual-Core
                        slot: Node 1 CPU 1
                        size: 3340MHz
                        capacity: 4GHz
                        width: 64 bits
                        clock: 133MHz

                        *-ide
                        description: IDE interface
                        product: CSB6 RAID/IDE Controller
                        vendor: Broadcom
                        physical id: f.1
                        bus info: pci@0000:00:0f.1
                        logical name: scsi3
                        version: a0
                        width: 32 bits
                        clock: 33MHz
                        capabilities: ide bus_master emulated
                        configuration: driver=pata_serverworks latency=64 module=pata_serverworks
                        *-cdrom
                        description: DVD reader
                        product: RW/DVD GCC-T10N
                        vendor: HL-DT-ST
                        physical id: 0.0.0
                        bus info: scsi@3:0.0.0
                        logical name: /dev/cdrom
                        logical name: /dev/cdwriter
                        logical name: /dev/dvd
                        logical name: /dev/scd0
                        logical name: /dev/sr0
                        version: 1.00
                        capabilities: removable audio cd-r cd-rw dvd
                        configuration: ansiversion=5 status=nodisc
                        *-storage
                        description: RAID bus controller
                        product: AAC-RAID
                        vendor: Adaptec
                        physical id: 2
                        bus info: pci@0000:01:02.0
                        logical name: scsi0
                        version: 02
                        width: 64 bits
                        clock: 66MHz
                        capabilities: storage pm msi pcix bus_master cap_list emulated
                        configuration: driver=aacraid latency=240 maxlatency=1 mingnt=1 module=aacraid
                        *-disk:0
                        description: SCSI Disk
                        product: Drive 1
                        vendor: IBM
                        physical id: 0.0.0
                        bus info: scsi@0:0.0.0
                        logical name: /dev/sda
                        version: V1.0
                        serial: C109DCE7
                        size: 68GB
                        capabilities: partitioned partitioned:dos
                        configuration: ansiversion=2
                        *-disk:1 UNCLAIMED
                        description: SCSI Disk
                        product: ST973402SS
                        vendor: IBM-ESXS
                        physical id: 1.0.0
                        bus info: scsi@0:1.0.0
                        version: B526
                        serial: 3NP1D1FS00009749ZHAG
                        capacity: 49GB
                        capabilities: 10000rpm
                        configuration: ansiversion=5
                        *-disk:2 UNCLAIMED
                        description: SCSI Disk
                        product: ST973401SS
                        vendor: IBM-ESXS
                        physical id: 1.1.0
                        bus info: scsi@0:1.1.0
                        version: B51D
                        serial: 3LB1MPM6000097365WR7
                        capacity: 68GB
                        capabilities: 10000rpm
                        configuration: ansiversion=5
                        *-disk:0
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.2
                        bus info: scsi@1:0.0.2
                        logical name: /dev/sdb
                        version: 0617
                        serial: SF64601404
                        size: 15GB
                        capacity: 15GB
                        capabilities: 7200rpm partitioned partitioned:dos
                        configuration: ansiversion=5
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.3
                        bus info: scsi@1:0.0.3
                        logical name: /dev/sdc
                        version: 0617
                        serial: SF64601404
                        size: 150GB
                        capacity: 150GB
                        capabilities: 7200rpm
                        configuration: ansiversion=5
                        *-disk:2
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.4
                        bus info: scsi@1:0.0.4
                        logical name: /dev/sdd
                        version: 0617
                        serial: SF64601404
                        size: 150GB
                        capacity: 150GB
                        capabilities: 7200rpm
                        configuration: ansiversion=5
                        *-disk:3
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.5
                        bus info: scsi@1:0.0.5
                        logical name: /dev/sde
                        version: 0617
                        serial: SF64601404
                        size: 150GB
                        capacity: 150GB
                        capabilities: 7200rpm
                        configuration: ansiversion=5
                        *-disk:4
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.7
                        -disk:22                                                                 THIS DISK IS HAVING SLOW IO PERFORMANCE PROBLEM*
                        description: SCSI Disk
                        product: 1726-4xx FAStT
                        vendor: IBM
                        physical id: 0.0.1d
                        bus info: scsi@1:0.0.29
                        logical name: /dev/sdx
                        version: 0617
                        serial: SF64601404
                        size: 931GB
                        capacity: 931GB
                        capabilities: 7200rpm
                        configuration: ansiversion=5



                        Server
                        IBM System X3950
                        Internal SAS Drive x 2
                        Qlogic 4Gbps HBA Card

                        SAN Specs
                        IBM DS-3400 SAS
                        IBM DS-3400 SATA
                        both are connected via HBA Switch and FC Card

                        Sorry I missed to mention about
                        in this same SAN Storage SAS Disks performance is very well_,
                        and we are facing IO Issue only with SATA Disk

                        all are configured with dm-multipath and
                        SAS Disks are mounted with OCFS2 on two servers
                        like RAW --> mpath --> OCFS2

                        SATA Disk is mounted with LVM
                        RAW --> MPATH --> LVM

                        Is it causes because of SATA disk Configured with Multipath & LVM?
                        Is it will resolve if we use RDAC Multipath Driver from IBM SDD for DS-3400 for this SATA Disk?

                        and also please suggest me a method to get High throughput from this SATA Disks.

                        Please help me.
                        • 9. Re: Slow IO Performance in Oracle uek 64bit kernel
                          Catch~22
                          Please contact Oracle support. Participation in this forum is on a voluntary basis. As such you are not in a position to request any official driver software or a solution from me for your HW configuration. I don't even have to reply to your post if I don't feel like it. If you need help for a production system, getting an Oracle Linux support subscription might be your best option.

                          Looking at your problem again, I wonder what records exist that point to an issue with the Oracle Linux kernel or driver. To find out what driver you are using, you can use the "lspci -v" command and filter out the information of the appropriate HBA controller.

                          If this is indeed a problem of using a 64-bit or 32-bit driver, I suggest to check the vendor support page (IBM) to see if there is a know issue. Perhaps your SAN or HBA controller or PC BIOS firmware requires a patch for 64-bit compatiblity. And again, to find out if it is a problem with the Oracle UEK kernel, simply boot the system using the RHEL kernel. A problem with the multipath configuration may causes the system to use a drive different than you expect, but afaik multipath does not provide the actual device driver to affect data throughput I/O, nor does LVM.