2 Replies Latest reply on Apr 4, 2012 1:00 AM by jschellSomeoneStoleMyAlias

    Unstable JVM


      In our environment we have:

      * OS - HP-UX 11i v31 Itanium ia64
      * IBM WAS –
      * Our production application JVM – single instance Application Server, min heap size 512M, max heap size 1024M
      * WAS java version (used by the app): Java(TM) 2 Runtime Environment, Standard Edition (IBM build 03_Dec_2009_08_45 hpux IA64W (SR7))

      Our production JVM, which runs a production application, becomes unusable periodically. When the issue occurs, it consumes lots of CPU and all connection attempts to it fail. It sometimes manages to sort things out and starts functioning properly. Nonetheless, it usually requires restart to get back to normal. To better understand the issue we enabled the Verbose Garbage Collection option (-Xverbosegc:file=) in our JVM. The output shows that when the problem occurs, the entire heap memory is used and the garbage collector works hard to release some of it. It executes full collections one by one and every takes long time (up to 15 seconds to complete). Memory usage graphs, based on the VGC output, show that there is a constant increase
      trend in memory usage since the start of the application. I may be wrong but this looks for me like a slow leak.

      When the issue starts, various different errors are observed in WAS logs. Sometimes we see “java.lang.OutOfMemoryError: Java heap space” (or similar), sometimes it says “Thread "WebContainer : " has been active for milliseconds and may be hung.” There is also a number of different errors, it’s hard to say which errors indicate the root cause of the issue and which are the effect of it.

      How can we diagnose the problem? We would be very gratefull if you can guide us through the issue. I don't expect anybody to fix the problem for me. I'd rather be told in which direction
      should I go, which tools should I use etc. I am definitely not a java expert, but have no other and have to fix it. Any suggestion, which can help me understand what is going on. is welcome.

        • 1. Re: Unstable JVM
          Try an IBM forum, or IBM support.
          • 2. Re: Unstable JVM
            user12006136 wrote:
            I am definitely not a java expert, but have no other and have to fix it. Any suggestion, which can help me understand what is going on. is welcome.
            So you have an OS, a VM, an Application (WAS) and then your company code that runs in there.

            And something isn't working.

            So you goal is to localize the problem.

            It would help if you can replicate it. That would require the following
            1. An install of everything some place besides production, like a test box.
            2. A way to simulate 'client' traffic - which is messages that look like the messages that already go into the production machine.
            3. Then you use 2 to send traffic for long periods of time to 1.
            4. At some point if 1 and 2 are right then you will get the same error.

            It would help if you can localize the problem. That is a lot easier if you can replicate it. If you can replicate then you can try removing things, generally parts of your company code. If you get to the point where that makes the problem go away then you might have a likely place for a problem (a bug in the code.)

            If you can't replicate it then you have to look at logs and hope you can find something strange. That is helped by experience.

            Depending on complexity of your company code this can take a long time (excluding just luck.)

            One possible solution, if your business allows for it, is to just automatically restart the VM periodically. This is to prevent it from becoming bad in the first place. Might not seem like an ideal solution but it is something that can be implemented quickly.