6 Replies Latest reply on Feb 5, 2013 3:37 PM by AigarsP

    OC Main Thread high swap usage

      Hi all!
      First, I hope this is correct forum to post, because my problem is with Weblogic+Oracle Service Bus using Jrockit
      (WLS 10.3.6+ OSB + JRocikt R28.2.4 + CentOs 6.2 64bit)

      So I got this problem, when frequently system becomes really slow for a while. Using top/iotop, I found that some threads are using high I/O and swap, specifically this:

      *"(OC Main Thread)" id=3 idx=0xc tid=35994 prio=5 alive, in native, daemon*

      And I get long regular GC pauses - around 10sec, and sometimes 30secs

      These are additional JVM parameters for server start:
      -Xms3072m -Xmx3072m -Xns128m -Xgc:pausetime -Xpausetarget=250ms -Djava.rmi.server.hostname=osb -Xmanagement:ssl=false,authenticate=false,autodiscovery=true,port=7099,rmiserver.port=7099 -XX:+UseCallProfiling

      Any suggestions and ideas would be appreciated
        • 1. Re: OC Main Thread high swap usage
          Kalyan Pasupuleti-Oracle

          Can you change these option.

          -Xns128m -Xgc:pausetime -Xpausetarget=250ms

          You defined new size only 128m out of 3Gb.

          I would suggest you keep atleast 1.25 GB and see the difference.

          • 2. Re: OC Main Thread high swap usage
            thanks for advice, I actually removed Xns flag, so far looks good, I will post more results after few days
            • 3. Re: OC Main Thread high swap usage
              Kalyan Pasupuleti-Oracle
              If you increase the value it will work more better way.

              I will tell you why.

              By default new size will take 1/4 of total heap size mean in 3 GB New size will be 768Mb so remaining will be for tenured one.

              If you increase new size then minor GC happen quite often which will cause any java thread blocked.

              So try to increase Xns value to see more better output.

              • 4. Re: OC Main Thread high swap usage
                I tried with 1.25G, GC are much less frequent, however they are 3s-11s long (without Xns average was 80ms, but very frequent young gc). Not sure if there is much bigger load on server or because of -Xns. I will play around once more without -Xns and post observations.
                Anyway, performance seems a lot better
                • 5. Re: OC Main Thread high swap usage
                  Kalyan Pasupuleti-Oracle
                  Thats really good to know.
                  • 6. Re: OC Main Thread high swap usage
                    got another performance problem again

                    now these are using almost 100% swap:

                    "(GC Worker Thread 1)" id=? idx=0x10 tid=50319 prio=5 alive, daemon

                    "(GC Worker Thread 2)" id=? idx=0x14 tid=50320 prio=5 alive, daemon

                    Problem is, when I restart weblogic, it works fast for some time, but at some moment, perfomance drops significally, and even if no services are being called, still IO wait% is high.

                    I tried different GC parameters, but in the end it slows down anyway.

                    Also, I found out that at some points there are some high IO on this process - could this mean anything?
                    "Thread-36" id=68 idx=0xf8 tid=50383 prio=5 alive, parked, native_blocked
                        -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x18bf4c698
                        at jrockit/vm/Locks.park0(J)V(Native Method)
                        at jrockit/vm/Locks.park(Locks.java:2230)
                        at sun/misc/Unsafe.park(ZJ)V(Native Method)
                        at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
                        at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                        at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                        at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
                        at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
                        at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:751)
                        at java/lang/Thread.run(Thread.java:662)
                        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
                        -- end of trace
                    otherwise, I am out of ideas where to look.
                    Any help is appreciated.