0 Replies Latest reply: Mar 9, 2010 3:33 AM by 843829 RSS

    JVM CRASH - out of swap space --- but swap space is available

    843829
      Hello,

      We have our application installed on Websphere Appliaction Server 5.1. We are having frequent JVM crashes. The application logs show no errors. WAS log file native_stdout.log show following error.

      Exception java.lang.OutOfMemoryError: requested 111976 bytes for promotion. Out of swap space?
      But there is a lot of swap space availabale -- 32G

      The GC seems to be working effectively as inferred from following excerpt from native_stdout file

      /*****

      Heap
      def new generation total 262144K, used 262144K [0x58800000, 0x6ac90000, 0x6ac90000)
      eden space 224704K, 100% used [0x58800000, 0x66370000, 0x66370000)
      from space 37440K, 100% used [0x68800000, 0x6ac90000, 0x6ac90000)
      to space 37440K, 0% used [0x66370000, 0x66370000, 0x68800000)
      tenured generation total 1797568K, used 1691983K [0x6ac90000, 0xd8800000, 0xd8800000)
      the space 1797568K, 94% used [0x6ac90000, 0xd20e3f28, 0xd20e4000, 0xd8800000)
      compacting perm gen total 131072K, used 88302K [0xd8800000, 0xe0800000, 0xf8800000)
      the space 131072K, 67% used [0xd8800000, 0xdde3b9c0, 0xdde3ba00, 0xe0800000)
      +8711.565: [DefNew: 262144K->37439K(262144K), 0.2357999 secs] 1954127K->1788025K(2059712K) Heap after GC invocations=338:+
      Heap
      def new generation total 262144K, used 37439K [0x58800000, 0x6ac90000, 0x6ac90000)
      eden space 224704K, 0% used [0x58800000, 0x58800000, 0x66370000)
      from space 37440K, 99% used [0x66370000, 0x687ffff8, 0x68800000)
      to space 37440K, 0% used [0x68800000, 0x68800000, 0x6ac90000)
      tenured generation total 1797568K, used 1750585K [0x6ac90000, 0xd8800000, 0xd8800000)
      the space 1797568K, 97% used [0x6ac90000, 0xd5a1e498, 0xd5a1e600, 0xd8800000)
      compacting perm gen total 131072K, used 88302K [0xd8800000, 0xe0800000, 0xf8800000)
      the space 131072K, 67% used [0xd8800000, 0xdde3b9c0, 0xdde3ba00, 0xe0800000)
      +} , 0.2365688 secs]+
      Total time for which application threads were stopped: 0.2406811 seconds
      Application time: 1.9636446 seconds
      +8713.777: [GC {Heap before GC invocations=338:+
      +Heap+
      +def new generation total 262144K, used 243685K [0x58800000, 0x6ac90000, 0x6ac90000)+
      +eden space 224704K, 91% used [0x58800000, 0x651695c0, 0x66370000)+
      +from space 37440K, 99% used [0x66370000, 0x687ffff8, 0x68800000)+
      +to space 37440K, 0% used [0x68800000, 0x68800000, 0x6ac90000)+
      +tenured generation total 1797568K, used 1750585K [0x6ac90000, 0xd8800000, 0xd8800000)+
      +the space 1797568K, 97% used [0x6ac90000, 0xd5a1e498, 0xd5a1e600, 0xd8800000)+
      +compacting perm gen total 131072K, used 88302K [0xd8800000, 0xe0800000, 0xf8800000)+
      +the space 131072K, 67% used [0xd8800000, 0xdde3ba30, 0xdde3bc00, 0xe0800000)+
      +8713.778: [DefNew+
      +Exception java.lang.OutOfMemoryError: requested 111976 bytes for promotion. Out of swap space?+
      +Application time: 0.5662442 seconds+
      +Total time for which application threads were stopped: 0.0007472 seconds+
      +Application time: 0.9423304 seconds+
      +Total time for which application threads were stopped: 0.0010760 seconds+

      +Application time: 0.1292099 seconds+
      *****/

      Following are the JVM enteries from server.xml

      <jvmEntries xmi:id="JavaVirtualMachine_1" verboseModeClass="false" verboseModeGarbageCollection="false" verboseModeJNI="false" i
      nitialHeapSize="2048" maximumHeapSize="2048" runHProf="false" debugMode="false" debugArgs="-Djava.compiler=NONE -Xdebug -Xnoagent -X
      runjdwp:transport=dt_socket,server=y,suspend=n,address=7777" genericJvmArguments="-server -Dcom.ibm.websphere.cookies.no.header=true
      -Dfile.encoding=UTF-8 -Dclient.encoding.override=UTF-8 -XX:CompileThreshold=1000 -XX:PermSize=128M -XX:MaxPermSize=512M -XX:Survivo
      rRatio=6 -XX:NewRatio=6 -XX:MaxLiveObjectEvacuationRatio=10 -XX:+DisableExplicitGC -Xbootclasspath/p:/apps/XYZ/WAS/WILY_8
      .0/connectors/v14.17/AutoProbeConnector.jar:/apps/XYZ/WAS/WILY_8.0/Agent.jar -Dcom.wily.introscope.agentProfile=/apps/XYZ/WAS/WILY_8.0/IntroscopeAgent_WAS5_MPHRO.profile" disableJIT="false">

      Recently we cahnge maxpermsize from 256M to 512M but that worsened the situation with more frequent crashes

      Can anybody please help me with this situation? Why JVM is crashing with out of swap space message when there is a lot of swap space available?

      Do we require any fix to be applied to WAS or JAVA? or tuning some WAS properties may be??

      Any help on this will be appreciated

      Thanks in Advance
      EKT