This discussion is archived
6 Replies Latest reply: Feb 5, 2013 7:37 AM by AigarsP RSS

OC Main Thread high swap usage

AigarsP Explorer
Currently Being Moderated
Hi all!
First, I hope this is correct forum to post, because my problem is with Weblogic+Oracle Service Bus using Jrockit
(WLS 10.3.6+ OSB 11.1.1.6. + JRocikt R28.2.4 + CentOs 6.2 64bit)

So I got this problem, when frequently system becomes really slow for a while. Using top/iotop, I found that some threads are using high I/O and swap, specifically this:

*"(OC Main Thread)" id=3 idx=0xc tid=35994 prio=5 alive, in native, daemon*

And I get long regular GC pauses - around 10sec, and sometimes 30secs

These are additional JVM parameters for server start:
-Xms3072m -Xmx3072m -Xns128m -Xgc:pausetime -Xpausetarget=250ms -Djava.rmi.server.hostname=osb -Xmanagement:ssl=false,authenticate=false,autodiscovery=true,port=7099,rmiserver.port=7099 -XX:+UseCallProfiling

Any suggestions and ideas would be appreciated
  • 1. Re: OC Main Thread high swap usage
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Hi,


    Can you change these option.

    -Xns128m -Xgc:pausetime -Xpausetarget=250ms

    You defined new size only 128m out of 3Gb.

    I would suggest you keep atleast 1.25 GB and see the difference.

    Regards,
    kal
  • 2. Re: OC Main Thread high swap usage
    AigarsP Explorer
    Currently Being Moderated
    thanks for advice, I actually removed Xns flag, so far looks good, I will post more results after few days
  • 3. Re: OC Main Thread high swap usage
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    If you increase the value it will work more better way.

    I will tell you why.

    By default new size will take 1/4 of total heap size mean in 3 GB New size will be 768Mb so remaining will be for tenured one.

    If you increase new size then minor GC happen quite often which will cause any java thread blocked.

    So try to increase Xns value to see more better output.

    Regards,
    Kal
  • 4. Re: OC Main Thread high swap usage
    AigarsP Explorer
    Currently Being Moderated
    I tried with 1.25G, GC are much less frequent, however they are 3s-11s long (without Xns average was 80ms, but very frequent young gc). Not sure if there is much bigger load on server or because of -Xns. I will play around once more without -Xns and post observations.
    Anyway, performance seems a lot better
  • 5. Re: OC Main Thread high swap usage
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Thats really good to know.
  • 6. Re: OC Main Thread high swap usage
    AigarsP Explorer
    Currently Being Moderated
    got another performance problem again

    now these are using almost 100% swap:

    "(GC Worker Thread 1)" id=? idx=0x10 tid=50319 prio=5 alive, daemon

    "(GC Worker Thread 2)" id=? idx=0x14 tid=50320 prio=5 alive, daemon

    Problem is, when I restart weblogic, it works fast for some time, but at some moment, perfomance drops significally, and even if no services are being called, still IO wait% is high.


    I tried different GC parameters, but in the end it slows down anyway.


    Also, I found out that at some points there are some high IO on this process - could this mean anything?
    "Thread-36" id=68 idx=0xf8 tid=50383 prio=5 alive, parked, native_blocked
        -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x18bf4c698
        at jrockit/vm/Locks.park0(J)V(Native Method)
        at jrockit/vm/Locks.park(Locks.java:2230)
        at sun/misc/Unsafe.park(ZJ)V(Native Method)
        at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
        at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
        at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
        at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:751)
        at java/lang/Thread.run(Thread.java:662)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    otherwise, I am out of ideas where to look.
    Any help is appreciated.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points