This discussion is archived
14 Replies Latest reply: Sep 5, 2012 2:24 AM by tbrinkmann RSS

System stuck on heavy workload / Harddisk

tbrinkmann Explorer
Currently Being Moderated
Hi,
we face a system gets stuck problem on heavy workload on local hard disk( virtual machine ) .

We run Oracle VM 2.2.2 with virtual machines running on Oracle Linux 5.8

If we create a heavy workload on the virtual machines by creating a archive with gz the hole
system gets stuck for that time.

I ´ve no idea where to find additional information.

Does anyone can help to work this out?

Thanks a LOT !
*T                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 1. Re: System stuck on heavy workload / Harddisk
    TommyReynolds Expert
    Currently Being Moderated
    we face a system gets stuck problem on heavy workload on local hard disk( virtual machine ) .
    Stuck how? One process? Can you log in while this is happening? Does it correct itself? How do you recover? What is the workload? I/O? Compute-bound?
    We run Oracle VM 2.2.2 with virtual machines running on Oracle Linux 5.8
    Are these PVM, HVM, or PVHVM guests? Are the disks directly handled by the domU guest? How are the disks presented to the servers? HBA? iSCSI? USB?
    If we create a heavy workload on the virtual machines by creating a archive with gz the hole system gets stuck for that time.
    On one guest? Multiple guests? Any guest? How do the other machines get "stuck".
    I ´ve no idea where to find additional information.
    First, check the Oracle VM User's Guide. There is actually some information in there.
    Does anyone can help to work this out?
    Not with the information we have so far.
  • 2. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Hi,
    the system ( VM Guest ) gets stuck. Database connection broke no login possible

    I actually not really sure if we use PVM but as I informed well the kernel must be the right one. We use

    2.6.18-274.0.0.0.1.el5xen

    The guest OS is placed on local HD`s but the gzip was created on a SAN LUN.

    We only have one guest system on that physikal host.

    Thanks for you help !
  • 3. Re: System stuck on heavy workload / Harddisk
    Dude! Guru
    Currently Being Moderated
    The xen kernel is a type 1 hypervisor. You would use such kernel if you use e.g. Oracle VM Server for x86. It is not meant to be used as a VM guest kernel, e.g. to run Oracle database. The PVM drivers are supposed to increase throughput of I/O operations in guest operating systems on top of the Oracle VM Server hosts.

    Edited by: Dude on Sep 5, 2012 2:11 AM

    Please see below.
  • 4. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Hmm ok,
    that may be a problem. But we use acfs2 and may correct me the UEK does not support acfs2 right ?

    We looking forward for the right kernel.

    *T                                                                                                                                                                                                                                                                                                                       
  • 5. Re: System stuck on heavy workload / Harddisk
    Avi Miller Guru
    Currently Being Moderated
    tbrinkmann wrote:
    that may be a problem. But we use acfs2 and may correct me the UEK does not support acfs2 right ?
    Do you mean OCFS2 or ACFS? They are different things, however both of them are supported by both the UEK and the UEK2. They are both developed by Oracle. :)
  • 6. Re: System stuck on heavy workload / Harddisk
    Avi Miller Guru
    Currently Being Moderated
    Dude wrote:
    The xen kernel is a type 1 hypervisor. You would use such kernel if you use e.g. Oracle VM Server for x86. It is not meant to be used as a VM guest kernel, e.g. to run Oracle database. The PVM drivers are supposed to increase throughput of I/O operations in guest operating systems on top of the Oracle VM Server hosts.
    No, the xen variant in the 2.6.18 family is the PVM kernel for DomU guests. It's not the hypervisor (that's Xen itself).
  • 7. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Ok. Thanks I understand the xen kernel :-/

    We use ACFS not ACFS2. We use it as a share for backup dpump and rman on both nodes.

    I start the UEK and try to start the acfs..

    #>uname -a
    Linux 2.6.32-200.13.1.el5uek #1 SMP Wed Jul 27 21:02:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

    #> acfsload start -s
    ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.32-200.13.1.el5ue

    Does anyone can help understand what I do wrong ?

    Thanks T*
  • 8. Re: System stuck on heavy workload / Harddisk
    Avi Miller Guru
    Currently Being Moderated
    tbrinkmann wrote:
    Does anyone can help understand what I do wrong ?
    Your Grid Infrastructure install is too old: you need 11.2.0.3 or higher for ACFS to work on 2.6.32-100, -200 and -300. Check My Oracle Support Doc ID 1369107.1 for more information.
  • 9. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Hey Avi
    thank you very much !

    We will update the grid up to PSU 1 or use the UEK .100 and we hope that will run now .-)

    Greetings *T                                                                                                                                                                                                                                                                                       
  • 10. Re: System stuck on heavy workload / Harddisk
    Dude! Guru
    Currently Being Moderated
    Sorry for causing confusion, my mistake. Both dom0 and paravirtualized domUs have to run the Xen kernel. (as Avi noticed)

    I think the experience that Gzip can starve other processes is a common problem. Gzip is generally CPU and I/O demanding. The symptom of loosing response time highly depends the availability of multiple CPU's and server specific hardware. So I'd say to answer the question will involve analyzing the hardware.

    To "solve" your problem, you could use the ionice tool to set program io scheduling class and priority, for instance:

    <pre>
    alias gzip='ionice -c 2 -n 7 nice -19 gzip'
    </pre>

    Then you can use "gzip" to run "niced" and "\gzip" at the command prompt to run gzip with real-time priority.

    When running "niced" the gzip process will still be on top of the list in the "top" utility, but as soon as you run another process, e.g. ls -lR /, the gzip process will move down the list with very low impact to the system.

    The man page "man ionice" has more information about running a program when the system is idle (c3), best effort (c2) or realtime (c1).

    Edited by: Dude on Sep 5, 2012 1:27 AM
  • 11. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Hey dude,
    we will keep that in mind. In the moment I try to install the UEK.100 suitable for our 11.2.0.3 grid home.
    I hope it will also a bedder way to run for an oracle cluster.

    I do face a problem by finding the older version of the kernel. I run my own repo I subscribed UEK_BASE and UEK_LATEST but there not the uek.100

    Now I will try to find it in older base installation media for OL may 5.6 or 5.7 ...

    Does this is the recommended way to install older kernel versions ? or is there a recommended way.

    Thanks *T                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 12. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Determine Oracle Linux Version From A Particular kernel-uek Version [ID 1441342.1]
    Red Hat and Oracle Linux Kernel Versions and Release Strings [ID 560992.1]
  • 13. Re: System stuck on heavy workload / Harddisk
    Dude! Guru
    Currently Being Moderated
    The Oracle Grid 11.2.0.3 distribution should have the necessary kernel driver to support ACFS under the UEK kernel. I'm not aware that you need a specific UEK verison. I'd rather use the latest UEK kernel.

    Or reconsider whether or not ACFS is the right choice for your environment, in particular if the disks are hardware RAID. ACFS is general purpose, but does not support any file type that can be directly stored in Oracle ASM. 11.2.0.3 adds ACFS support for data pump dumpsets, for instance.

    Anyway, I guess you've moved on to solve an installation issue, which is a different topic.
  • 14. Re: System stuck on heavy workload / Harddisk
    tbrinkmann Explorer
    Currently Being Moderated
    Re dude,
    you are right but keep in mind we do not installed a single PSU :-/ we are only using basic installation of
    11.2.0.3 and the meta doc 1369107.1 show´s that the UEK for this GRID_HOME version is .100 if we want to use .300 we need the latest PSU 11.2.0.3.*2*

    We close this discussion here and may start a new one on installation.
    Thanks *T                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points