13 Replies Latest reply: Oct 30, 2012 7:30 AM by 998899-Oracle RSS

    Problem with weblogic 11g out of memory

    998899-Oracle
      Hi,


      I am using WLS 10.3.6 with JRockit . I am a newbie working with WLS

      I keep facing an issue where the machine(16gb ram and 500gb HDD ) were weblogic is deployed keep getting out of memory .. I only used the default configuration for the weblogic deployment that came with the weblogic installation .

      There is only one adf application running on the weblogic which is accessed by around two to three people ..

      What additional changes needs to be done?

      Second question is that is there a way to find how many DB connections are open currently in the weblogic and how long its been open?

      What debug option can be used to find the reason for out of memory erro?

      Thanks in advance,
      Sam
        • 1. Re: Problem with weblogic 11g out of memory
          Kalyan Pasupuleti-Oracle
          Hi Sam,

          By default weblogic will give min heap as -Xms256m and max heap as -Xmx512m and if you are using Sun JDK it will use MaxPermsize as 128m.


          Now to avoid such OOOM (Out of Memory) issue try to increase these parameters.


          If you are using 64 bit I would suggest to use 2GB Minimum or 8Gb MaX so ideal one would be like this.

          -Xms3072m -Xmx3072m MaxPermSize=1024m

          If you are using 32 bit then we cannot increase more than 2 GB there.


          Now coming to Connection with DB is purely depend on how many Max connection pool size had set there with in Datasource.

          Datasource --> Connection Pool -->


          Initial capacity and Max pool size and increment size.


          So try to set max pool according to your requirement by default it would be 15 Max and 1 initial capacity and 1 for increment.

          Now for third option you need not keep any further debug simply check the logs to find out the exact cause of the OOM issue and collect heap dump and gc logs.

          Regards,
          Kal
          • 2. Re: Problem with weblogic 11g out of memory
            998899-Oracle
            Hi Kalyan,


            Thanks for the reply. How are these parameters value calculated ? Are they dependent on one another??


            -Sam
            • 3. Re: Problem with weblogic 11g out of memory
              René van Wijk
              As you are using JRockit, you can use mission control to get more information about the why the out-of-memory error is occurring.
              For example, you can run a flight recording and monitor the live data on the heap. If you see the live data constantly rising, it might
              be smart to run the memory leak detector, with which you can pinpoint the culprit. Examples of how to use the flight recording
              and memory leak detector are provided here: http://middlewaremagic.com/weblogic/?p=6930.

              In order to find out, how many database connections are currently open, you can use the monitoring tab of your datasource.
              Note that initially it not showing much information, this can be changed by clicking on the 'edit table' link. Here you have attributes
              such 'active connections...', Active Connections Current Count: The number of connections currently in use by applications.

              If you suspect the application to be leaking connections, you can also set a value higher than zero to the Inactive Connection Timeout
              "The number of inactive seconds on a reserved connection before WebLogic Server reclaims the connection and releases it back into the connection pool.
              You can use the Inactive Connection Timeout feature to reclaim leaked connections - connections that were not explicitly closed by the application. Note
              that this feature is not intended to be used in place of properly closing connections. When set to 0, the feature is disabled."
              This parameter is present on the connection pool, configuration tab of your data source.

              What you also can try to do is disable the statement caching, by setting the statement cache size to zero. Statement Cache Size     
              "The number of prepared and callable statements stored in the cache. (This may increase server performance.) WebLogic Server can reuse statements
              in the cache without reloading the statements, which can increase server performance. Each connection in the connection pool has its own cache of statements.
              Setting the size of the statement cache to 0 turns off statement caching."
              Note that the database itself caches statements as well.

              We also had a similar issue with the statement caching on WebLogic and ADF, which also ran into out-of-memory errors. By turning of the statement caching
              on the WebLogic side the issue (in our case) got resolved.
              • 4. Re: Problem with weblogic 11g out of memory
                998899-Oracle
                Hi Rene,

                Thanks for the reply . When i tried to connect to the remote server using mission control, I was unable to connect even after proving the following parameters in the java options .

                Xmanagement:ssl=false,authenticate=false,port=7099 Djava.rmi.server.hostname=hostAddress(where weblogic is located).

                And how do i get the heap dumb and garbage collection logs before the server crashed ?

                Since i am new to this . What are the things i should be looking for to detect the problem ? How long should a normal flight be run ? Does running it for a long time increase jvm memory utilization too ?


                Thanks
                Sam
                • 5. Re: Problem with weblogic 11g out of memory
                  Kalyan Pasupuleti-Oracle
                  Hi,

                  Majorly when you running Mission control try to take look for GC graph and Heap size.


                  These two are major thing to verify through online monitoring.

                  to check later you can take JRF recording to analyze them later for better clarification.

                  Regards,
                  Kal
                  • 6. Re: Problem with weblogic 11g out of memory
                    998899-Oracle
                    Hi Kal,

                    Thanks for the reply.


                    But i am not able to connect to remote machine using JRockit Mission Control I gave the necessary settings in the java options still its not connection.

                    Xmanagement:ssl=false,authenticate=false,port=7099 Djava.rmi.server.hostname=10.177.223.24(where weblogic is located).

                    Even then it says unable to connect

                    -Sam
                    • 7. Re: Problem with weblogic 11g out of memory
                      René van Wijk
                      The parameters (-Djava.rmi.server.hostname=hostname -Xmanagement:ssl=false,authenticate=false,port=7099) you entered look correct.

                      Is there a firewall between the server from which you try to connect and the server where WebLogic is running? If that is the case you
                      have to make sure the port you defined is able to pass through the firewall (On Linux this means you have to edit the IPtables).

                      Also (just to double check if the parameters are picked) can run a ps command (if you are on Linux), for example, ps -ef|grep YourServerName.
                      Check the output to see if the parameters are present.

                      "And how do i get the heap dumb and garbage collection logs before the server crashed ?"
                      With JRockit you can add the option -XX:+|-HeapDumpOnOutOfMemoryError (http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABBBEAJ).
                      This will produce a dump of the Java heap in HPROF binary format, which can be analyzed by using for example memory analyzer (http://www.eclipse.org/mat/)

                      JRockit also provides the -XX:+|-HeapDiagnosticsOnOutOfMemoryError (http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABBBEAJ)
                      which prints Java heap diagnostics when an out-of-memory error occurs.

                      " Does running it for a long time increase jvm memory utilization too ?"
                      Luckily the flight recording is very light weigth with a very small overhead.

                      "How long should a normal flight be run ?"
                      When your system is under load, let it run for about 30 minutes or so.

                      "What are the things i should be looking for to detect the problem ?"
                      When you open the flight recording the general CPU usage will open. When you click on the heap usage tab, you will see a blue line (probably going up
                      and down) that shows the heap usage. There is also a green line that will show the amount of live data on the heap. When this line keeps
                      growing you are probably dealing with a memory leak. In the memory, gc graph you will see a similar graph (but with the addition of bars)
                      that show the pause times introduced for a particular collection. Here also the same green line (live data) is present. When you notice the green
                      line is only ascending you are probably dealing dealing with a memory leak in the application. To determine where the memory leak is coming from
                      you can use the memory leak detector (an example of its usage can be found here: http://middlewaremagic.com/weblogic/?p=6930).
                      • 8. Re: Problem with weblogic 11g out of memory
                        998899-Oracle
                        Hi,

                        One More thing . I have specified in the setDomainEnv.sh
                        USER_MEM_ARGS="-Xms6144m -Xmx6144m -XgcPrio:throughput"
                        
                        . ${WL_HOME}/common/bin/commEnv.sh
                        
                        WLS_HOME="${WL_HOME}/server"
                        export WLS_HOME
                        
                        XMS_SUN_64BIT="256"
                        export XMS_SUN_64BIT
                        XMS_SUN_32BIT="256"
                        export XMS_SUN_32BIT
                        XMX_SUN_64BIT="512"
                        export XMX_SUN_64BIT
                        XMX_SUN_32BIT="512"
                        export XMX_SUN_32BIT
                        XMS_JROCKIT_64BIT="3072"
                        export XMS_JROCKIT_64BIT
                        XMS_JROCKIT_32BIT="256"
                        export XMS_JROCKIT_32BIT
                        XMX_JROCKIT_64BIT="3072"
                        export XMX_JROCKIT_64BIT
                        XMX_JROCKIT_32BIT="512"
                        export XMX_JROCKIT_32BIT
                        
                        
                        if [ "${JAVA_VENDOR}" = "Sun" ] ; then
                             WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"
                             export WLS_MEM_ARGS_64BIT
                             WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"
                             export WLS_MEM_ARGS_32BIT
                        else
                             WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"
                             export WLS_MEM_ARGS_64BIT
                             WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"
                             export WLS_MEM_ARGS_32BIT
                        fi
                        
                        if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
                             CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_JROCKIT_64BIT}m -Xmx${XMX_JROCKIT_64BIT}m"
                             export CUSTOM_MEM_ARGS_64BIT
                             CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_JROCKIT_32BIT}m -Xmx${XMX_JROCKIT_32BIT}m"
                             export CUSTOM_MEM_ARGS_32BIT
                        else
                             CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_SUN_64BIT}m -Xmx${XMX_SUN_64BIT}m"
                             export CUSTOM_MEM_ARGS_64BIT
                             CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_SUN_32BIT}m -Xmx${XMX_SUN_32BIT}m"
                             export CUSTOM_MEM_ARGS_32BIT
                        fi
                        
                        
                        MEM_ARGS_64BIT="${CUSTOM_MEM_ARGS_64BIT}"
                        export MEM_ARGS_64BIT
                        
                        MEM_ARGS_32BIT="${CUSTOM_MEM_ARGS_32BIT}"
                        export MEM_ARGS_32BIT
                        
                        if [ "${JAVA_USE_64BIT}" = "true" ] ; then
                             MEM_ARGS="${MEM_ARGS_64BIT}"
                             export MEM_ARGS
                        else
                             MEM_ARGS="${MEM_ARGS_32BIT}"
                             export MEM_ARGS
                        fi
                        
                        MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"
                        export MEM_PERM_SIZE_64BIT
                        
                        MEM_PERM_SIZE_32BIT="-XX:PermSize=128m"
                        export MEM_PERM_SIZE_32BIT
                        
                        if [ "${JAVA_USE_64BIT}" = "true" ] ; then
                             MEM_PERM_SIZE="${MEM_PERM_SIZE_64BIT}"
                             export MEM_PERM_SIZE
                        else
                             MEM_PERM_SIZE="${MEM_PERM_SIZE_32BIT}"
                             export MEM_PERM_SIZE
                        fi
                        
                        MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=1024m"
                        export MEM_MAX_PERM_SIZE_64BIT
                        
                        MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=512m"
                        export MEM_MAX_PERM_SIZE_32BIT
                        
                        if [ "${JAVA_USE_64BIT}" = "true" ] ; then
                             MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_64BIT}"
                             export MEM_MAX_PERM_SIZE
                        else
                             MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_32BIT}"
                             export MEM_MAX_PERM_SIZE
                        fi
                        
                        if [ "${JAVA_VENDOR}" = "Sun" ] ; then
                             if [ "${PRODUCTION_MODE}" = "" ] ; then
                                  MEM_DEV_ARGS="-XX:CompileThreshold=8000 ${MEM_PERM_SIZE} "
                                  export MEM_DEV_ARGS
                             fi
                        fi
                        
                        # Had to have a separate test here BECAUSE of immediate variable expansion on windows
                        
                        if [ "${JAVA_VENDOR}" = "Sun" ] ; then
                             MEM_ARGS="${MEM_ARGS} ${MEM_DEV_ARGS} ${MEM_MAX_PERM_SIZE}"
                             export MEM_ARGS
                        fi
                        
                        if [ "${JAVA_VENDOR}" = "HP" ] ; then
                             MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
                             export MEM_ARGS
                        fi
                        
                        if [ "${JAVA_VENDOR}" = "Apple" ] ; then
                             MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
                             export MEM_ARGS
                        fi
                        
                        # IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values
                        
                        if [ "${USER_MEM_ARGS}" != "" ] ; then
                             MEM_ARGS="${USER_MEM_ARGS}"
                             export MEM_ARGS
                        fi
                        How can i find out if it has picked up the right memory argument ? So what could be the argument ? I saw that USER_MEM_ARGS has been defined ? so is that the memory argument value ?

                        Thanks
                        -Sam
                        • 9. Re: Problem with weblogic 11g out of memory
                          Ganesh..
                          During server startup, ${MEM_ARGS} and ${JAVA_OPTIONS} along with some other jvm options are displayed on the console.

                          echo "Starting WLS with line:"
                          +echo "${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}"+


                          You can check if your changes are picked up.
                          • 10. Re: Problem with weblogic 11g out of memory
                            René van Wijk
                            Using USER_MEM_ARGS will override the MEM_ARGS set by the script:
                            if [ "${USER_MEM_ARGS}" != "" ] ; then
                                 MEM_ARGS="${USER_MEM_ARGS}"
                                 export MEM_ARGS
                            fi
                            When you want to be sure the management arguments are picked up you can use something like
                            if [ "${SERVER_NAME}" = "AdminServer" ] ; then
                                 # JMX configuration for JRockit
                                    JMX_OPTIONS="-Djava.rmi.server.hostname=192.168.1.150 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
                                    export JMX_OPTIONS
                                    # Memory arguments for JRockit
                                    USER_MEM_ARGS="-jrockit -Xms512m -Xmx512m -Xns128m -XXkeepAreaRatio:25 -Xgc:pausetime -XpauseTarget:200ms -XX:+UseCallProfiling -XX:+UseLargePagesForHeap ${JMX_OPTIONS}"
                                    export USER_MEM_ARGS
                            fi
                            To complete call of scripts when you use, for example, startWebLogic (in $DOMAIN_HOME/bin) is
                            - startWebLogic.sh, which calls setDomainEnv.sh, which calls commEnv.sh (located in the WL_HOME/common/bin directory). The last script sets up the (native) libraries and classpath.

                            The line that will start a weblogic looks as follows:
                            ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
                            By adding the JMX_OPTIONS (shown above) to the USER_MEM_ARGS you are always save, as this overrides the MEM_ARGS.
                            • 11. Re: Problem with weblogic 11g out of memory
                              998899-Oracle
                              Hi Rene,

                              Thanks for the reply. So according to your previous replies and going through blog , These are conclusions i have made

                              My weblogic was running on the following configuration so far : -Xms6144m -Xmx6144m -XgcPrio:throughput having 5gb as heap space and a point to be noted it was used very heavily but just one or two users time to time

                              I ran jstat <pid> 1 1
                              HeapSize         6291456.0
                              NurserySize      2939588.9
                              UsedHeapSize   1840865.3
                              YC                   16 
                              OC                   8
                              YCTime            6.296
                              OCTime            14.534
                              GCTime            20.831
                              YCPauseTime    6.297
                              OCPauseTime    7.127
                              PauseTime        13.424
                              I would like to use throughput instead of pausetime so i would like to go with a configuration something similar to this:
                              if [ "${SERVER_NAME}" = "AdminServer" ] ; then
                                       # JMX configuration for JRockit
                                      JMX_OPTIONS="-Djava.rmi.server.hostname=10.177.220.52 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
                                      export JMX_OPTIONS
                                      # Memory arguments for JRockit
                                      USER_MEM_ARGS="-jrockit -Xms3072m -Xmx3072m -xns128m -XXkeepAreaRatio:25 -Xgc:throughput -XX:+UseCallProfiling -XX:+UseLargePagesForHeap -XX:+|-HeapDiagnosticsOnOutOfMemoryError ${JMX_OPTIONS}"
                                      export USER_MEM_ARGS
                              fi
                              On the furthur reading through the blog i found that we need to set the following for logging ?
                              -Xverbose:opt,memory,gcpause,memdbg,compaction,gc 
                              -Xverbosetimestamp 
                              -Xverboselog:/var/opt/weblogic/weblogic/10.3/user_projects/Domain/servers/jvm_log.log
                              So my configuration would finally look like this
                              if [ "${SERVER_NAME}" = "AdminServer" ] ; then
                                       # JMX configuration for JRockit
                                      JMX_OPTIONS="-Djava.rmi.server.hostname=10.177.220.52 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
                                      export JMX_OPTIONS
                                      # Memory arguments for JRockit
                                      USER_MEM_ARGS="-jrockit -Xms3072m -Xmx3072m -xns128m -XXkeepAreaRatio:25 -Xgc:throughput -XX:+UseCallProfiling -XX:+UseLargePagesForHeap -XX:+|-HeapDiagnosticsOnOutOfMemoryError -Xverbose:opt,memory,gcpause,memdbg,compaction,gc -Xverbosetimestamp -Xverboselog:/var/opt/weblogic/weblogic/10.3/user_projects/Domain/servers/jvm_log.log ${JMX_OPTIONS}"
                                      export USER_MEM_ARGS
                              fi
                              Any extra parameters to be added or anything that seems unwanted ?

                              such as -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+UseParallelOldGC -XX:+AggressiveOpts -XX:+UseBiasedLocking ?

                              Thanks,
                              Sam
                              • 12. Re: Problem with weblogic 11g out of memory
                                René van Wijk
                                "Any extra parameters to be added or anything that seems unwanted ? such as -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+UseParallelOldGC -XX:+AggressiveOpts -XX:+UseBiasedLocking ?"

                                Note that these parameters are for the Sun HotSpot JVM. Instead of AggresiveOpts you could use -XXaggressive (http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABBEBEC). For the ParallelGCThreads you can use the option -XXgcThreads (http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABFHBJJ), by default, the number of threads are based on the number of cores and hardware threads on the machine.

                                JRockit in general does not need much tuning, a lot comes out of the box when selecting a certain garbage collection scheme such as throughput. Such as, for example, the setting -XXkeepAreaRatio:25 is the default (http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABIHJFH) so you can leave that out if you want.

                                Below you can find an example:
                                if [ "${SERVER_NAME}" = "AdminServer" ] ; then
                                     # JMX configuration for JRockit
                                        JMX_OPTIONS="-Djava.rmi.server.hostname=10.177.220.52 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
                                        export JMX_OPTIONS
                                        
                                        # LOGGING options
                                        LOGGING_OPTIONS="-Xverbose:opt,memory,gcpause,memdbg,compaction,gc -Xverbosetimestamp -Xverboselog:/var/opt/weblogic/weblogic/10.3/user_projects/Domain/servers/jvm_log.log"
                                        export LOGGING_OPTIONS
                                        
                                        # DUMP OPTIONS
                                        DUMP_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/opt/weblogic/weblogic/10.3/user_projects/Domain/servers/"
                                        export DUMP_OPTIONS
                                
                                        # Memory arguments for JRockit
                                        USER_MEM_ARGS="-jrockit -Xms3072m -Xmx3072m -Xns1024m -Xgc:throughput -XX:+UseCallProfiling -XX:+UseLargePagesForHeap ${DUMP_OPTIONS} ${LOGGING_OPTIONS} ${JMX_OPTIONS}"
                                        export USER_MEM_ARGS
                                fi
                                Note that in order to use -XX:+UseLargePagesForHeap you have to configure large pages in the OS, the steps involved are presented here: http://middlewaremagic.com/weblogic/?p=8133 (which also show more info on how to tweak the operating system, when running a Java EE Server such as WebLogic).

                                Another thing you might want to look into is the command-line interface JRockit offers. By using jrcmd (located in the ${JROCKIT_HOME}/bin directory) you can also start a flight recording, for example,

                                ./jrcmd <pid> start_flightrecording name=SomeRecording settings=${JROCKIT_HOME}/jre/lib/jfr/profile.jfs duration=30m filename=/some/directory/SomeRecording.jfr

                                To check the recording you can use

                                ./jrcmd <pid> check_flightrecording

                                To stop the recording you can use

                                ./jrcmd <pid> stop_flightrecording name=SomeRecording (or you can use the ID of the recording that is obtained by the check_flightrecording command: recording=<ID>)
                                • 13. Re: Problem with weblogic 11g out of memory
                                  998899-Oracle
                                  Hi Rene,

                                  Thanks for correcting me. This was the log that was generated after restarting the server and setting the java options .One more thing it shows that the changes made for JRockit Mission control is picked up in the server options but still i could not connect it through my local Jrockit kit . I might have to check the firewall settings i guess.
                                  [WARN ] Use of -Djrockit.optfile is deprecated and discouraged.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Minimum TLA size is 2048 bytes.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Preferred TLA size is 131072 bytes.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] TLA waste limit is 2048 bytes.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Nursery object limit max is 131072 bytes.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Nursery object limit percentage is 1.0%.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Running with 32 bit heap and compressed references.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Java heap regions:
                                   [000000003fffe000 - 000000007fffe000] rw-- (pagesize=0x1000)
                                   [0000000080000000 - 0000000100000000] rw-- (pagesize=0x1000)
                                   3GB committed, 0B reserved.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Initial and maximum number of gc threads: 4, of which 4 parallel threads, 2 concurrent threads, and 4 yc threads.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Maximum nursery percentage of free heap is 95%.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Setting keepAreaMarkers[0] to 0x5fffe3d8.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Setting keepAreaMarkers[1] to 0x6fffe5c8.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Setting keepAreaMarkers[2] to 0x80000800.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Next keeparea will start at 0x6fffe5c8 and end at 0x80000800.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Nursery size increased from 0KB to 1048578KB. Nursery list consists of 2 parts.
                                  [nursery][Tue Oct 30 08:49:38 2012][22417] Average part size: 524289KB. Contraharmonic mean (CHM): 1048573KB. CHM per part: 524287KB. Normalized CHM: 0.999995.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Prefetch distance in workpacket: 4. Packet size: 493 objects.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] GC mode: Garbage collection optimized for throughput, strategy: Generational Parallel Mark & Sweep.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Heap size: 3145728KB, maximal heap size: 3145728KB, nursery size: 1048578KB.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <start>-<end>: <type> <before>KB-><after>KB (<heap>KB), <time> ms, sum of pauses <pause> ms.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <start>  - start time of collection (seconds since jvm start).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <type>   - OC (old collection) or YC (young collection).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <end>    - end time of collection (seconds since jvm start).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <before> - memory used by objects before collection (KB).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <after>  - memory used by objects after collection (KB).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <heap>   - size of heap after collection (KB).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <time>   - total time of collection (milliseconds).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] <pause>  - total sum of pauses during collection (milliseconds).
                                  [memory ][Tue Oct 30 08:49:38 2012][22417]            Run with -Xverbose:gcpause to see individual phases.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Using prefetch linesize: 64 bytes chunks: 512 bytes pf_dist: 512 bytes.
                                  [memory ][Tue Oct 30 08:49:38 2012][22417] Running on hypervisor Unknown (but probably running on a hypervisor)
                                  [mgmnt  ][Tue Oct 30 08:49:39 2012][22417] Remote JMX connector started at address igbu-apps:7091 
                                  [mgmnt  ][Tue Oct 30 08:49:39 2012][22417] Local JMX connector started
                                  [mgmnt  ][Tue Oct 30 08:49:39 2012][22417] JDP autodiscovery started at address 232.192.1.212:7095
                                  [memory ][Tue Oct 30 08:50:13 2012][22417] Adding 3 final handles from dying thread 'process reaper'.
                                  [memory ][Tue Oct 30 08:50:13 2012][22417] Adding 3 final handles from dying thread 'process reaper'.
                                  [memory ][Tue Oct 30 08:50:19 2012][22417] Adding 29459 final handles from dying thread 'JMX FRAMEWORK Domain Runtime MB'.
                                  [memory ][Tue Oct 30 08:50:22 2012][22417] Adding 3 final handles from dying thread 'process reaper'.
                                  [memory ][Tue Oct 30 08:50:22 2012][22417] Adding 3 final handles from dying thread 'process reaper'.
                                  [memory ][Tue Oct 30 08:50:36 2012][22417] [YC#1] GC reason: Allocation request failed.
                                  [memory ][Tue Oct 30 08:50:36 2012][22417] [YC#1] 57.950: YC started.
                                  [alloc  ][Tue Oct 30 08:50:36 2012][22417] [YC#1] Pending requests at 'Before YC' - Total: 1, TLAs: 1 (approx 131072 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [nursery][Tue Oct 30 08:50:36 2012][22417] [YC#1] Young collection 1 started. This YC is running while the OC is in phase: not running.
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] SemiRef phase WeakJNIHandles run in single threaded mode.
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] SemiRef phase ClassConstraints run in single threaded mode.
                                  [nursery][Tue Oct 30 08:50:37 2012][22417] [YC#1] Next keeparea will start at 0x5fffe3d8 and end at 0x6fffe5c8.
                                  [alloc  ][Tue Oct 30 08:50:37 2012][22417] [YC#1] Satisfied 0 object and 1 tla allocations. Pending requests went from 1 to 0.
                                  [alloc  ][Tue Oct 30 08:50:37 2012][22417] [YC#1] Pending requests at 'After YC' - Total: 0, TLAs: 0 (approx 0 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [gcpause][Tue Oct 30 08:50:37 2012][22417] [YC#1] [---]   701.553 ms (57.950000-58.652000)  YC
                                  [gcpause][Tue Oct 30 08:50:37 2012][22417] [YC#1] [con]     0.025 ms (57.950000-57.951000)  YC:PreGC
                                  [gcpause][Tue Oct 30 08:50:37 2012][22417] [YC#1] [pau]   701.445 ms (57.951000-58.652000)  YC:Main
                                  [gcpause][Tue Oct 30 08:50:37 2012][22417] [YC#1] [con]     0.008 ms (58.652000-58.652000)  YC:PostGC
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] YC promoted 1372770 objects (121314KB).
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] Page faults before YC: 1, page faults after YC: 1, pages in heap: 786432.
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] Nursery size after YC: 1048578KB. (Free: 786431KB Parts: 2)
                                  [memory ][Tue Oct 30 08:50:37 2012][22417] [YC#1] 57.950-58.652: YC 1110104KB->451751KB (3145728KB), 0.702 s, sum of pauses 701.445 ms, longest pause 701.445 ms.
                                  [opt    ][Tue Oct 30 08:50:47 2012][22417][00036] #1 (Opt) jrockit/vm/RNI.callProfilerHelper(IJI)V
                                  [opt    ][Tue Oct 30 08:50:47 2012][22417][00036] #1 68.574-68.577 0x7fa38b61e000-0x7fa38b61e0a0 3.46 ms 256KB 33574 bc/s (3.46 ms 33574 bc/s)
                                  [memory ][Tue Oct 30 08:50:55 2012][22417] [YC#2] GC reason: Allocation request failed.
                                  [memory ][Tue Oct 30 08:50:55 2012][22417] [YC#2] 77.077: YC started.
                                  [alloc  ][Tue Oct 30 08:50:55 2012][22417] [YC#2] Pending requests at 'Before YC' - Total: 1, TLAs: 1 (approx 131072 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [nursery][Tue Oct 30 08:50:55 2012][22417] [YC#2] Young collection 2 started. This YC is running while the OC is in phase: not running.
                                  [memory ][Tue Oct 30 08:50:55 2012][22417] [YC#2] SemiRef phase WeakJNIHandles run in single threaded mode.
                                  [memory ][Tue Oct 30 08:50:56 2012][22417] [YC#2] SemiRef phase ClassConstraints run in single threaded mode.
                                  [nursery][Tue Oct 30 08:50:56 2012][22417] [YC#2] Next keeparea will start at 0x6fffe5c8 and end at 0x80000800.
                                  [alloc  ][Tue Oct 30 08:50:56 2012][22417] [YC#2] Satisfied 0 object and 1 tla allocations. Pending requests went from 1 to 0.
                                  [alloc  ][Tue Oct 30 08:50:56 2012][22417] [YC#2] Pending requests at 'After YC' - Total: 0, TLAs: 0 (approx 0 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [gcpause][Tue Oct 30 08:50:56 2012][22417] [YC#2] [---]   583.980 ms (77.077000-77.661000)  YC
                                  [gcpause][Tue Oct 30 08:50:56 2012][22417] [YC#2] [con]     0.023 ms (77.077000-77.077000)  YC:PreGC
                                  [gcpause][Tue Oct 30 08:50:56 2012][22417] [YC#2] [pau]   583.859 ms (77.077000-77.661000)  YC:Main
                                  [gcpause][Tue Oct 30 08:50:56 2012][22417] [YC#2] [con]     0.014 ms (77.661000-77.661000)  YC:PostGC
                                  [memory ][Tue Oct 30 08:50:56 2012][22417] [YC#2] YC promoted 867858 objects (66096KB).
                                  [memory ][Tue Oct 30 08:50:56 2012][22417] [YC#2] Page faults before YC: 1, page faults after YC: 1, pages in heap: 786432.
                                  [memory ][Tue Oct 30 08:50:56 2012][22417] [YC#2] Nursery size after YC: 1048578KB. (Free: 786429KB Parts: 2)
                                  [memory ][Tue Oct 30 08:50:56 2012][22417] [YC#2] 77.077-77.661: YC 1246139KB->531195KB (3145728KB), 0.584 s, sum of pauses 583.859 ms, longest pause 583.859 ms.
                                  [opt    ][Tue Oct 30 08:50:58 2012][22417][00036] #2 (Opt) jrockit/vm/Allocator.allocObjectOrArray(JJIZ)Ljava/lang/Object;
                                  [opt    ][Tue Oct 30 08:50:58 2012][22417][00036] #2 80.275-80.280 0x7fa38b61e0a0-0x7fa38b61e280 5.17 ms 256KB 56530 bc/s (8.62 ms 47329 bc/s)
                                  [memory ][Tue Oct 30 08:51:12 2012][22417] Adding 4 final handles from dying thread 'Timer-3'.
                                  [memory ][Tue Oct 30 08:51:22 2012][22417] [YC#3] GC reason: Allocation request failed.
                                  [memory ][Tue Oct 30 08:51:22 2012][22417] [YC#3] 104.291: YC started.
                                  [alloc  ][Tue Oct 30 08:51:22 2012][22417] [YC#3] Pending requests at 'Before YC' - Total: 1, TLAs: 1 (approx 131072 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [nursery][Tue Oct 30 08:51:22 2012][22417] [YC#3] Young collection 3 started. This YC is running while the OC is in phase: not running.
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] SemiRef phase WeakJNIHandles run in single threaded mode.
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] SemiRef phase ClassConstraints run in single threaded mode.
                                  [nursery][Tue Oct 30 08:51:23 2012][22417] [YC#3] Next keeparea will start at 0x5fffe3d8 and end at 0x6fffe5c8.
                                  [alloc  ][Tue Oct 30 08:51:23 2012][22417] [YC#3] Satisfied 0 object and 1 tla allocations. Pending requests went from 1 to 0.
                                  [alloc  ][Tue Oct 30 08:51:23 2012][22417] [YC#3] Pending requests at 'After YC' - Total: 0, TLAs: 0 (approx 0 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [gcpause][Tue Oct 30 08:51:23 2012][22417] [YC#3] [---]   485.838 ms (104.291000-104.776000)  YC
                                  [gcpause][Tue Oct 30 08:51:23 2012][22417] [YC#3] [con]     0.033 ms (104.291000-104.291000)  YC:PreGC
                                  [gcpause][Tue Oct 30 08:51:23 2012][22417] [YC#3] [pau]   485.726 ms (104.291000-104.776000)  YC:Main
                                  [gcpause][Tue Oct 30 08:51:23 2012][22417] [YC#3] [con]     0.012 ms (104.776000-104.776000)  YC:PostGC
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] YC promoted 1773485 objects (88438KB).
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] Page faults before YC: 1, page faults after YC: 1, pages in heap: 786432.
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] Nursery size after YC: 1048578KB. (Free: 786431KB Parts: 2)
                                  [memory ][Tue Oct 30 08:51:23 2012][22417] [YC#3] 104.291-104.776: YC 1351910KB->656486KB (3145728KB), 0.486 s, sum of pauses 485.726 ms, longest pause 485.726 ms.
                                  [memory ][Tue Oct 30 08:51:36 2012][22417] [YC#4] GC reason: Allocation request failed.
                                  [memory ][Tue Oct 30 08:51:36 2012][22417] [YC#4] 118.171: YC started.
                                  [alloc  ][Tue Oct 30 08:51:36 2012][22417] [YC#4] Pending requests at 'Before YC' - Total: 1, TLAs: 1 (approx 131072 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [nursery][Tue Oct 30 08:51:36 2012][22417] [YC#4] Young collection 4 started. This YC is running while the OC is in phase: not running.
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] SemiRef phase WeakJNIHandles run in single threaded mode.
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] SemiRef phase ClassConstraints run in single threaded mode.
                                  [nursery][Tue Oct 30 08:51:37 2012][22417] [YC#4] Next keeparea will start at 0x6fffe5c8 and end at 0x80000800.
                                  [alloc  ][Tue Oct 30 08:51:37 2012][22417] [YC#4] Satisfied 0 object and 1 tla allocations. Pending requests went from 1 to 0.
                                  [alloc  ][Tue Oct 30 08:51:37 2012][22417] [YC#4] Pending requests at 'After YC' - Total: 0, TLAs: 0 (approx 0 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [gcpause][Tue Oct 30 08:51:37 2012][22417] [YC#4] [---]   527.888 ms (118.171000-118.699000)  YC
                                  [gcpause][Tue Oct 30 08:51:37 2012][22417] [YC#4] [con]     0.030 ms (118.171000-118.171000)  YC:PreGC
                                  [gcpause][Tue Oct 30 08:51:37 2012][22417] [YC#4] [pau]   527.746 ms (118.172000-118.699000)  YC:Main
                                  [gcpause][Tue Oct 30 08:51:37 2012][22417] [YC#4] [con]     0.014 ms (118.699000-118.699000)  YC:PostGC
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] YC promoted 359792 objects (17750KB).
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] Page faults before YC: 1, page faults after YC: 1, pages in heap: 786432.
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] Nursery size after YC: 1048578KB. (Free: 786429KB Parts: 2)
                                  [memory ][Tue Oct 30 08:51:37 2012][22417] [YC#4] 118.171-118.699: YC 1452530KB->684647KB (3145728KB), 0.528 s, sum of pauses 527.746 ms, longest pause 527.746 ms.
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] GC reason: Allocation request failed.
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] 157.978: YC started.
                                  [alloc  ][Tue Oct 30 08:52:16 2012][22417] [YC#5] Pending requests at 'Before YC' - Total: 1, TLAs: 1 (approx 131072 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [nursery][Tue Oct 30 08:52:16 2012][22417] [YC#5] Young collection 5 started. This YC is running while the OC is in phase: not running.
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] SemiRef phase WeakJNIHandles run in single threaded mode.
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] SemiRef phase ClassConstraints run in single threaded mode.
                                  [nursery][Tue Oct 30 08:52:16 2012][22417] [YC#5] Next keeparea will start at 0x5fffe3d8 and end at 0x6fffe5c8.
                                  [alloc  ][Tue Oct 30 08:52:16 2012][22417] [YC#5] Satisfied 0 object and 1 tla allocations. Pending requests went from 1 to 0.
                                  [alloc  ][Tue Oct 30 08:52:16 2012][22417] [YC#5] Pending requests at 'After YC' - Total: 0, TLAs: 0 (approx 0 bytes), objects: 0 (0 bytes). Max age: 0.
                                  [gcpause][Tue Oct 30 08:52:16 2012][22417] [YC#5] [---]   159.662 ms (157.978000-158.137000)  YC
                                  [gcpause][Tue Oct 30 08:52:16 2012][22417] [YC#5] [con]     0.023 ms (157.978000-157.978000)  YC:PreGC
                                  [gcpause][Tue Oct 30 08:52:16 2012][22417] [YC#5] [pau]   159.461 ms (157.978000-158.137000)  YC:Main
                                  [gcpause][Tue Oct 30 08:52:16 2012][22417] [YC#5] [con]     0.017 ms (158.137000-158.137000)  YC:PostGC
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] YC promoted 1314197 objects (65091KB).
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] Page faults before YC: 1, page faults after YC: 1, pages in heap: 786432.
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] Nursery size after YC: 1048578KB. (Free: 786431KB Parts: 2)
                                  [memory ][Tue Oct 30 08:52:16 2012][22417] [YC#5] 157.978-158.137: YC 1519122KB->800813KB (3145728KB), 0.160 s, sum of pauses 159.461 ms, longest pause 159.461 ms.
                                  Thanks
                                  -Sam