0 Replies Latest reply: Apr 26, 2012 7:43 AM by 933114 RSS

    Repeating OOM GC overhead limit exceeded after free memory recovery

    933114
      h3. Environment.
      Application server (JBoss) running on the Linux 64-bit VM with params
      -server -Xms3072m -Xmx3072m -XX:MaxPermSize=196m -XX:+DisableExplicitGC -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=4 -XX:SoftRefLRUPolicyMSPerMB=200 -XX:+UseCompressedOops -Dfile.encoding=ISO-8859-1

      h3. Problem.
      One of the threads executing business logic overloaded memory for near 19 minutes and finally failed with OOM (GC overhead limit exceeded). - it is not a problem.
      *OOM GC overhead limit exceeded repeated at the application server level after free memory recovery!* - What is a cause of such behavior ??

      h3. Incident details.

      Time Logged OOM cases
      09:54     1
      09:55     1
      09:56     1
      09:57     2
      09:58     4
      09:59     4
      10:00     4
      10:01     5
      10:02     6
      10:03     65
      10:04     87
      10:05     109
      10:06     129
      10:07     149
      10:08     169
      10:09     189
      10:10     201

      Time Free memory amount
      9:44:00     1306.32
      9:45:00     129.29
      9:47:00     146.6
      9:48:00     350.81
      9:49:00     342.33
      9:50:00     346.96
      9:51:00     369.89
      9:52:00     483.96
      10:01:00     342.54
      10:03:00     1971.41

      There were no full GC between 10:03 and 10:10.

      OOM appeared again and again without stacktrace during the latest occurence. All OOM occurences were logged to the STDOUT. So there were not handled by both application and JBoss.

      OOM was logged with an exception stack during the previous similar incident. This exception stacktrace block belongs to the stateful bean response custom serialization by application (application specific code is eliminated - be sure in it is stable at usual conditions).
      at java.util.Arrays.copyOf(Arrays.java:2882)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
      at java.lang.StringBuilder.append(StringBuilder.java:119)
      at java.io.ObjectStreamClass.getClassSignature(ObjectStreamClass.java:1457)
      at java.io.ObjectStreamField.<init>(ObjectStreamField.java:120)
      at java.io.ObjectStreamClass.getDefaultSerialFields(ObjectStreamClass.java:1596)
      at java.io.ObjectStreamClass.getSerialFields(ObjectStreamClass.java:1512)
      at java.io.ObjectStreamClass.access$800(ObjectStreamClass.java:52)
      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:427)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
      at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      at java.util.ArrayList.readObject(ArrayList.java:593)
      at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
      at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1666)
      Edited by: Alexey.Svatseff on 26.04.2012 7:42