1 Reply Latest reply on Nov 17, 2009 8:48 AM by 843798

    JVM threw OutOfMemory during gc

      I am running jre 1.6.0_10 on linux. I have configured my JVM as follws:
      -server -Xms5g -Xmx5g -Xss256k -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseParallelOldGC -XX:+UseTLAB -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:-UseGCOverheadLimit -Xloggc:/var/log/admob/ustore/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
      . My JVM crashed and left behind the gc traces. I saw that none of the generation is fully occupied at the time of the crash. I don't understand why JVM crashed when there is plenty memory left. I've tried -XX:-UseGCOverheadLimit option (as suggested by http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#par_gc.oom) with no success. Does anyone know what else could be contributing to this OOM?

      Here is the gc log at the time of the crash:
      2009-09-23T01:21:19.785+0000: 114547.407: [GC
      Desired survivor size 177864704 bytes, new threshold 1 (max 15)
       [PSYoungGen: 1901930K->154280K(1922304K)] 3835892K->2092341K(5068032K), 0.4399220 secs] [Times: user=2.36 sys=0.23, real=0.45 secs] 
      2009-09-23T01:21:50.247+0000: 114577.870: [GC
      Desired survivor size 177602560 bytes, new threshold 1 (max 15)
       [PSYoungGen: 1902312K->155297K(1923712K)] 3840373K->2097412K(5069440K), 0.4261560 secs] [Times: user=2.24 sys=0.24, real=0.43 secs] 
      2009-09-23T01:22:22.564+0000: 114610.186: [GC
      Desired survivor size 176488448 bytes, new threshold 1 (max 15)
       [PSYoungGen: 1905313K->153516K(1923456K)] 3847428K->2099785K(5069184K), 0.2816380 secs] [Times: user=1.24 sys=0.08, real=0.28 secs] 
      2009-09-23T01:22:53.514+0000: 114641.136: [GC
      Desired survivor size 175439872 bytes, new threshold 1 (max 15)
       [PSYoungGen: 1903532K->153540K(1925824K)] 3849801K->2103848K(5071552K), 0.4128470 secs] [Times: user=2.13 sys=0.25, real=0.41 secs] 
      2009-09-23T01:23:23.218+0000: 114670.840: [GC
      Desired survivor size 173998080 bytes, new threshold 1 (max 15)
       [PSYoungGen: 1907012K->152522K(1924800K)] 3857320K->2106836K(5070528K), 0.4092880 secs] [Times: user=2.18 sys=0.18, real=0.41 secs] 
      2009-09-23T01:23:35.499+0000: 114683.121: [GC
      Desired survivor size 192348160 bytes, new threshold 1 (max 15)
       [PSYoungGen: 558998K->34442K(1891392K)] 2513312K->1998089K(5037120K), 0.1063600 secs] [Times: user=0.55 sys=0.07, real=0.10 secs] 
      2009-09-23T01:23:35.606+0000: 114683.228: [GC
      Desired survivor size 211615744 bytes, new threshold 1 (max 15)
       [PSYoungGen: 34442K->0K(1721536K)] 1998089K->1998576K(4867264K), 0.0772960 secs] [Times: user=0.23 sys=0.03, real=0.08 secs] 
      2009-09-23T01:23:35.683+0000: 114683.305: [Full GC [PSYoungGen: 0K->0K(1721536K)] [ParOldGen: 1998576K->115455K(3145728K)] 1998576K->115455K(4867264K) [PSPermGen: 19904K->19822K(40256K)], 0.7404460 secs] [Times: user=2.26 sys=0.09, re
      al=0.74 secs] 
      2009-09-23T01:23:36.424+0000: 114684.046: [GC
      Desired survivor size 214695936 bytes, new threshold 1 (max 15)
       [PSYoungGen: 0K->0K(1884480K)] 115455K->115455K(5030208K), 0.0089010 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
      2009-09-23T01:23:36.433+0000: 114684.056: [Full GC [PSYoungGen: 0K->0K(1884480K)] [ParOldGen: 115455K->111179K(3145728K)] 115455K->111179K(5030208K) [PSPermGen: 19822K->19814K(44992K)], 0.4016820 secs] [Times: user=1.30 sys=0.00, real
      =0.40 secs] 
       PSYoungGen      total 1884480K, used 82198K [0x00002aab73650000, 0x00002aabf3650000, 0x00002aabf3650000)
        eden space 1677824K, 4% used [0x00002aab73650000,0x00002aab786958b0,0x00002aabd9cd0000)
        from space 206656K, 0% used [0x00002aabe6c80000,0x00002aabe6c80000,0x00002aabf3650000)
        to   space 209664K, 0% used [0x00002aabd9cd0000,0x00002aabd9cd0000,0x00002aabe6990000)
       ParOldGen       total 3145728K, used 111179K [0x00002aaab3650000, 0x00002aab73650000, 0x00002aab73650000)
        object space 3145728K, 3% used [0x00002aaab3650000,0x00002aaaba2e2d20,0x00002aab73650000)
       PSPermGen       total 44992K, used 19861K [0x00002aaaae250000, 0x00002aaab0e40000, 0x00002aaab3650000)
        object space 44992K, 44% used [0x00002aaaae250000,0x00002aaaaf5b5760,0x00002aaab0e40000)
        • 1. Re: JVM threw OutOfMemory during gc
          An OOME can indicate a shortage of memory in several areas. These could be common to GC algorithms across major JVM vendors Sun,IBM, BEA

          (#1) Tenured generation:     This would mean "heap". We know memory is segmented into many genations. But as and when the tenured generation gets full and cannot be expanded any further, the JVM considers itself OOM
          (#2) Permanant generation: This does not resize during the life time of the application, regardless of how much free space may exist in the rest of the heap, but remains at whatever it was originally set to (default is 64K). Should this prove too small for the permanant generation, then JVM will throw an OOME even if there's plenty of heap left. Adding the -XX:+PrintHeapAtGC switch will tell you if this is the case.
          (#3) The third possibility is your OS (Linux in this case) is out of memory, e.g. you've asked for a 2GB heap on a box with 1GB RAM and 512MB swap space
          (#4) Another possibility is native components are hogging your 4GB ceiling. Native code competes with the JVM to use the 4GB of addressable space in your application. If these components are memory hungry, your app will be starved of addressable space, even if it hasn't actually used up all the heap you've given it yet.

          I am hoping one of these four could be possible for your JVM to throw OOME. I recently had similar problems on Windows 2003 SP2 and the reason was made out to be possibility #3. Let me know