7 Replies Latest reply on Feb 29, 2012 2:35 AM by Dude!

    disk write I/O capped at 3MB/s with OEL 5.7

    Anyao-Oracle
      I downloaded OEL 5.7 from edelivery, and installed it on machine "tt" with a 1TB disk (7200RPM). A "dd" test showed that the disk write I/O is around 3MB/s. "uname -a" on machine "tt" returns "Linux tt *2.6.18-274.el5xen* #1 SMP Mon Jul 25 14:24:57 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux".

      dd if=/dev/zero of=/tmp/test bs=1M count=100 oflag=dsync
      100+0 records in
      100+0 records out
      104857600 bytes (105 MB) copied, 38.3419 seconds, 2.7 MB/s

      When we tried the same command on machine "dhoodpc" with the same 1TB disk (same model, same everything), we got around 100MB/s consistently across multiple runs. This machine "dhoodpc" has OEL 5.3 installed. "uname -a" returns "Linux dhoodpc *2.6.18-128.el5* #1 SMP Wed Jan 21 08:45:05 EST 2009 x86_64 x86_64 x86_64 GNU/Linux".

      dd if=/dev/zero of=/ds0/test bs=1M count=100 oflag=dsync
      100+0 records in
      100+0 records out
      104857600 bytes (105 MB) copied, 0.963998 seconds, 109 MB/s

      Question #1:
      I noticed the kernel version on OEL 5.7 is a "xen" kernel, versus the kernel version of OEL 5.3 is a regular kernel. Is "xen" kernel the default in OEL 5.7? I followed the default setting while installing OEL 5.7.

      Is the "xen" kernel the reason why I got only 3MB/s disk write I/O on OEL 5.7?

      Question #2:
      "hdparm" showed the "using_dma" is off for the disk on machine "tt", versus it's "on" for the disk on machine "dhoodpc". When I tried to enable it through "hdparm -d1 /dev/hdc", I got an error. My questions are:
      1) is the setting of "using_dma=off" a reason of the slow disk performance?
      2) if yes, how can I switch it on?


      The hdparm shows that the disk on machine "tt" has using_dma off.

      [root@tt~]# hdparm /dev/hdc
      /dev/hdc:
      multcount = 16 (on)
      IO_support = 0 (default 16-bit)
      unmaskirq = 0 (off)
      using_dma    =  0 (off)
      keepsettings = 0 (off)
      readonly = 0 (off)
      readahead = 256 (on)
      geometry = 65535/255/63, sectors = 3907029168, start = 0      hdparm /dev/hde

      [root@dhoodpc ~]# hdparm /dev/hde
      /dev/hde:
      multcount = 16 (on)
      IO_support = 0 (default 16-bit)
      unmaskirq = 0 (off)
      using_dma    =  1 (on)
      keepsettings = 0 (off)
      readonly = 0 (off)
      readahead = 256 (on)
      geometry = 60801/255/63, sectors = 976773168, start = 0

      Then I tried to switch using_dma on but got the following error.
      [root@tt~]# hdparm -d1 /dev/hdc
      /dev/hdc:
      setting using_dma to 1 (on)
      HDIO_SET_DMA failed: Operation not permitted
        • 1. Re: disk write I/O capped at 3MB/s with OEL 5.7
          Avi Miller-Oracle
          anyao wrote:
          Question #1:
          I noticed the kernel version on OEL 5.7 is a "xen" kernel, versus the kernel version of OEL 5.3 is a regular kernel. Is "xen" kernel the default in OEL 5.7? I followed the default setting while installing OEL 5.7.
          No, the default kernel for OL5.7 is the UEK (2.6.32) kernel, not the 2.6.18xen variant. You should switch your boot to the 2.6.32 or 2.6.18 non-Xen kernel and test again.
          Is the "xen" kernel the reason why I got only 3MB/s disk write I/O on OEL 5.7?
          Possibly.
          Question #2:
          "hdparm" showed the "using_dma" is off for the disk on machine "tt", versus it's "on" for the disk on machine "dhoodpc". When I tried to enable it through "hdparm -d1 /dev/hdc", I got an error. My questions are:
          1) is the setting of "using_dma=off" a reason of the slow disk performance?
          Possibly.
          2) if yes, how can I switch it on?
          Switch to a non-Xen kernel and try it again.
          • 2. Re: disk write I/O capped at 3MB/s with OEL 5.7
            Anyao-Oracle
            Thanks for your quick reply, Avi.

            Unfortunately I don't see other optional kernels to boot from. I chose all default configuration settings for OEL 5.7 installation. It seems the installer only installed the xen kernel. Am I missing anything here? Is there a special configuration I should choose at installation?

            [root@tt boot]# ls
            config-2.6.18-274.el5xen System.map-2.6.18-274.el5xen
            grub vmlinuz-2.6.18-274.el5xen
            initrd-2.6.18-274.el5xen.img xen.gz-2.6.18-274.el5
            symvers-2.6.18-274.el5xen.gz xen-syms-2.6.18-274.el5

            [root@tt grub]# cat menu.lst
            # grub.conf generated by anaconda
            #
            # Note that you do not have to rerun grub after making changes to this file
            # NOTICE: You do not have a /boot partition. This means that
            # all kernel and initrd paths are relative to /, eg.
            # root (hd0,0)
            # kernel /boot/vmlinuz-version ro root=/dev/hda1
            # initrd /boot/initrd-version.img
            #boot=/dev/hda
            default=0
            timeout=5
            splashimage=(hd0,0)/boot/grub/splash.xpm.gz
            hiddenmenu
            title Oracle Linux Server (2.6.18-274.el5xen)
            root (hd0,0)
            kernel /boot/xen.gz-2.6.18-274.el5
            module /boot/vmlinuz-2.6.18-274.el5xen ro root=LABEL=/ rhgb quiet
            module /boot/initrd-2.6.18-274.el5xen.img
            title Other
            rootnoverify (hd1,0)
            chainloader +1
            • 3. Re: disk write I/O capped at 3MB/s with OEL 5.7
              Avi Miller-Oracle
              anyao wrote:
              Unfortunately I don't see other optional kernels to boot from. I chose all default configuration settings for OEL 5.7 installation. It seems the installer only installed the xen kernel. Am I missing anything here?
              How did you do the install? If you install from the ISO, the default kernel is kernel-uek with a normal kernel installed as well (2.6.32 and 2.6.18) and the Xen kernel is not installed. So, I have no idea how you got the Xen kernel if you did a normal ISO install.
              • 4. Re: disk write I/O capped at 3MB/s with OEL 5.7
                Anyao-Oracle
                I downloaded OEL 5.7 from edelivery, copied the ISO binary to a bootle usb drive, and install it from the usb drive.

                Edelivery says the OEL 5.7 binary (V27570-01.zip) was last updated on Jan 03, 2012, but the extracted ISO file OracleLinux-R5-U7-Server-x86_64-dvd.iso has a date of 7/29/2011 1:58:18AM 3.58GB. Is this expected?

                You seem to suggest that there is an option in the installation wizard to install xen kernel rather than the normal uek kernel. Which option is it? I don't think I picked up any special option about kernel type. Do you think installing from usb drive (other than a dvd) makes any difference here?

                Also, rather than destroying and reinstalling, is there a way to "upgrade" the current installation and add the normal uek kernel so that we can change the grub and boot that kernel instead?
                • 5. Re: disk write I/O capped at 3MB/s with OEL 5.7
                  Avi Miller-Oracle
                  anyao wrote:
                  You seem to suggest that there is an option in the installation wizard to install xen kernel rather than the normal uek kernel. Which option is it? I don't think I picked up any special option about kernel type. Do you think installing from usb drive (other than a dvd) makes any difference here?
                  The Xen kernel is (or should only be) installed when you're doing a PVM installation under a Xen hypervisor. I have no idea why your USB-based install is selecting this kernel instead of the default kernel-uek which is what is installed when I do a 5.7 install from the ISO. I would be very interested to know if the same thing happens if you install from the ISO "properly", i.e. without using the USB device. We don't do any testing of installation from USB media.
                  Also, rather than destroying and reinstalling, is there a way to "upgrade" the current installation and add the normal uek kernel so that we can change the grub and boot that kernel instead?
                  Yes, just install the kernel-uek RPMs using yum or up2date.
                  • 6. Re: disk write I/O capped at 3MB/s with OEL 5.7
                    Anyao-Oracle
                    After installing and booting from the uek kernel, the disk performs at around 80MB/s. Thanks a lot for your prompt help, Avi.
                    • 7. Re: disk write I/O capped at 3MB/s with OEL 5.7
                      Dude!
                      Btw, you can also test performance using hdparm. The following, for instance, is the output I get when OL 5.7 as Guest OS in Virtualbox:
                      [root@vm014 ~]# hdparm -tT /dev/sda
                      
                      /dev/sda:
                       Timing cached reads:   21332 MB in  2.00 seconds = 10679.78 MB/sec
                       Timing buffered disk reads:  342 MB in  3.02 seconds = 113.36 MB/sec
                      Regarding DMA, DMA allows reading and writing data without CPU involvement, thus making the processor available for other tasks. The performance advantage can be significant. Unfortunately I do not have a system with IDE drives, only SATA, which do not have a DMA setting.

                      From what I understand, a Xen kernel does not support DMA and all the memory addresses are remapped by the virtual machine software. An input/output memory management unit (IOMMU) handles this remapping, allowing for the native device drivers to be used in a guest operating system. This is sometimes called PCI passthrough and requires CPU virtualization support (AMD-V or Intel-VT)

                      From what I understand the difference IOMMU versus DMA causes some moderate overhead, but it should not make such a hugh difference like 100 MB/s versus 3 MB/s, unless perhaps your CPU does not support hardware virtualization or the driver in the XEN kernel requires an update.

                      Other factors that can have a significant impact with hard disks of the same brand and type beside the CPU and driver are the drive's firmware.