1 2 Previous Next 15 Replies Latest reply: May 8, 2008 12:38 PM by 843829 RSS

    unexpected OutOfMemoryError

    843829
      Hello!

      We are currently developing a model checker for micro controllers at the University of Aachen. While running a rather large example for evaluation of a new feature we got the following error message:
      #
      # An unexpected error has been detected by Java Runtime Environment:
      #
      # java.lang.OutOfMemoryError: requested -1610612736 bytes for GrET in C:\build_area\jdk6_10\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
      #
      #  Internal Error (allocation.inline.hpp:42), pid=5092, tid=2556
      #  Error: GrET in C:\build_area\jdk6_10\hotspot\src\share\vm\utilities\growableArray.cpp
      #
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b11 mixed mode windows-amd64)
      # An error report file with more information is saved as:
      # C:\Users\volker.kamin\Documents\mc_square\hs_err_pid5092.log
      #
      # If you would like to submit a bug report, please visit:
      #   http://java.sun.com/webapps/bugreport/crash.jsp
      #
      java.lang.OutOfMemoryError
      The VM arguments are:
      -server -Xmx240G -Xss100M
      and
      -server -Xmx240G -Xss100M -XX:+UseParallelOldGC

      The server is a SunFire X4600 with 256GB of RAM.

      The negative allocation size seems to be some kind of internal overflow. What can we do?
      The physical RAM seems not to be the problem, since the JRE does only allocate around 160GB of RAM.

      Sincerely
      Volker Kamin
        • 1. Re: unexpected OutOfMemoryError
          843829
          VolkerKamin wrote:
          The VM arguments are:
          -server -Xmx240G -Xss100M
          and
          -server -Xmx240G -Xss100M -XX:+UseParallelOldGC
          Try setting -Xms and -Xmx to the same value, and see if the JVM starts (this will make the JVM start by allocating the maximum amount of memory). Also, do you really need a 100M stack?
          • 2. Re: unexpected OutOfMemoryError
            843829
            paul.miner wrote:
            Try setting -Xms and -Xmx to the same value, and see if the JVM starts (this will make the JVM start by allocating the maximum amount of memory). Also, do you really need a 100M stack?
            I dont think I need the stack. I changed the value to -Xss 20M
            I set -Xms and -Xmx to 190G (currently there is another job running and 50GB are already allocated). When I now start nothing special happens. The OS is Windows Server Enterprise 2007 SP1. The JVM is Sun's Java 1.6.0_10 (64bit).

            I am rerunning right now. Results might take a little, as multi-core support is currently developed.
            • 3. Re: unexpected OutOfMemoryError
              843829
              It helps to set the options to equal values, but that does not explain what happened.
              Is it possible, that Hotspot is somehow limitted to the number of times it can allocate new memory? I am really confused about this.
              • 4. Re: unexpected OutOfMemoryError
                843829
                VolkerKamin wrote:
                It helps to set the options to equal values, but that does not explain what happened.
                Is it possible, that Hotspot is somehow limitted to the number of times it can allocate new memory? I am really confused about this.
                I suspect it has something to do with the need for the heap space to be one contiguous block of address space, and that this address space probably can't move during the lifetime of the JVM. Probably the address space near the heap gets allocated for non-heap purposes, in turn limiting how much the heap can grow.

                I do know I've run into this problem with the 32-bit JVM, so I've made it a habit of always configuring server JVMs with the same Xmx and Xms values.

                About the negative allocation in the error message, maybe you should submit a bug report and suggest that the code that generates that error message is limited to 32-bit values?
                • 5. Re: unexpected OutOfMemoryError
                  843829
                  Do I understand this correct?

                  The JRE needs a continuous block of memory for its heap? So once another program allocated memory at the end of the JRE memory, the JRE cannot grow anymore???
                  • 6. Re: unexpected OutOfMemoryError
                    843829
                    VolkerKamin wrote:
                    Do I understand this correct?

                    The JRE needs a continuous block of memory for its heap? So once another program allocated memory at the end of the JRE memory, the JRE cannot grow anymore???
                    That is my understanding, yes. Note that it is the address space of the heap that must be contiguous, the physical memory (or page file) backing it need not be contiguous.
                    • 7. Re: unexpected OutOfMemoryError
                      843829
                      Sorry. That's a bug in our code. I know exactly where it is and there's already a bug filed for it, but I'm at JavaOne this week and don't have 5 minutes together at a time to find the bug ID. I have a fix, too. (Actually, a fix for this immediate problem and an even better fix that builds a more appropriate data structure that won't have this problem at all.) If it's any consolation, the guy who wrote that code no longer works for us. :-)

                      I think the problem is related to the shape of your data structures in the heap, but there's not much you can do about that. And I wouldn't want you to!
                      • 8. Re: unexpected OutOfMemoryError
                        843829
                        paul.miner wrote:
                        VolkerKamin wrote:
                        Do I understand this correct?

                        The JRE needs a continuous block of memory for its heap? So once another program allocated memory at the end of the JRE memory, the JRE cannot grow anymore???
                        That is my understanding, yes. Note that it is the address space of the heap that must be contiguous, the physical memory (or page file) backing it need not be contiguous.
                        It's true the Sun's JVM (HotSpot) uses a single contiguous region of virtual addresses for the java heap. However, the virtual addresses for the maximum size of the java heap are reserved at JVM startup; once reserved other libraries should not interfere with them. (Note that other programs can never interfere with those mappings, each program (process) has its own virtual address space).
                        After initialization, the JVM then allocates memory to back the reserved virtual addresses as needed to grow the heap.
                        • 9. Re: unexpected OutOfMemoryError
                          843829
                          jxc wrote:
                          paul.miner wrote:
                          VolkerKamin wrote:
                          Do I understand this correct?

                          The JRE needs a continuous block of memory for its heap? So once another program allocated memory at the end of the JRE memory, the JRE cannot grow anymore???
                          That is my understanding, yes. Note that it is the address space of the heap that must be contiguous, the physical memory (or page file) backing it need not be contiguous.
                          It's true the Sun's JVM (HotSpot) uses a single contiguous region of virtual addresses for the java heap. However, the virtual addresses for the maximum size of the java heap are reserved at JVM startup; once reserved other libraries should not interfere with them. (Note that other programs can never interfere with those mappings, each program (process) has its own virtual address space).
                          After initialization, the JVM then allocates memory to back the reserved virtual addresses as needed to grow the heap.
                          So is the bug that PeterKessler mentioned related to this reservation being disrupted by the process itself? Or was he trying to explain a different source of the problem?
                          • 10. Re: unexpected OutOfMemoryError
                            843829
                            paul.miner wrote:
                            So is the bug that PeterKessler mentioned related to this reservation being disrupted by the process itself? Or was he trying to explain a different source of the problem?
                            He's talking about a different source of the problem; the reservation isn't being disrupted. The bug he mentioned for the immediate problem is an overflow when calculating how much c-heap memory to allocate for a data structure used by the JVM. I found the bug id; see [bug 6588262|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6588262].

                            -John
                            • 11. Re: unexpected OutOfMemoryError
                              843829
                              jxc wrote:
                              paul.miner wrote:
                              So is the bug that PeterKessler mentioned related to this reservation being disrupted by the process itself? Or was he trying to explain a different source of the problem?
                              He's talking about a different source of the problem; the reservation isn't being disrupted. The bug he mentioned for the immediate problem is an overflow when calculating how much c-heap memory to allocate for a data structure used by the JVM. I found the bug id; see [bug 6588262|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6588262].

                              -John
                              So did setting Xms and Xmx to the same value make the problem disappear (so far) because it didn't need to grow the JVM structure?
                              • 12. Re: unexpected OutOfMemoryError
                                843829
                                paul.miner wrote:
                                So did setting Xms and Xmx to the same value make the problem disappear (so far) because it didn't need to grow the JVM structure?
                                I suspect it just delayed the problem by delaying GCs. The data structure we think is involved is a marking stack, which doesn't necessarily grow linearly with the heap. Having the full error log (hs_err<pid>.log) would help verify it; it includes the native frame stack trace which we can decode.

                                Volker, if you have the full error log, can you post it or send it to hotspot-gc-use at openjdk dot java dot net?
                                • 13. Re: unexpected OutOfMemoryError
                                  843829
                                  Pardon the threadjack, but I submitted a completely reproducible bug in the Java 6 HotSpot compiler, received a review id, and never heard anything further about it from Sun. I've also talked to a couple other people with similar symptoms but who were unable to reduce their problems to a test case (we all had largish apps). How can I find out what's going on with this? I posted the report here: http://www.blinkenbyte.org/jvm/hotspot_bug_reviewid_1085121.html The only thing I'd add at this point is that in addition -Xint, disabling native compilation on the method in question would also make the problem disappear, but the nature of the problem is worrying in itself (random deadlock in perfectly valid code).

                                  EDIT: Thank you for getting that bug report going again: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6699669

                                  Edited by: paul.miner on May 9, 2008 11:00 AM
                                  • 14. Re: unexpected OutOfMemoryError
                                    843829
                                    Hi!

                                    here the hs_err_pid5092.log
                                    #
                                    # An unexpected error has been detected by Java Runtime Environment:
                                    #
                                    # java.lang.OutOfMemoryError: requested -1610612736 bytes for GrET in C:\build_area\jdk6_10\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
                                    #
                                    #  Internal Error (allocation.inline.hpp:42), pid=5092, tid=2556
                                    #  Error: GrET in C:\build_area\jdk6_10\hotspot\src\share\vm\utilities\growableArray.cpp
                                    #
                                    # Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b11 mixed mode windows-amd64)
                                    # If you would like to submit a bug report, please visit:
                                    #   http://java.sun.com/webapps/bugreport/crash.jsp
                                    #
                                    
                                    ---------------  T H R E A D  ---------------
                                    
                                    Current thread (0x000000000087f000):  GCTaskThread [stack: 0x00000000024a0000,0x00000000025a0000] [id=2556]
                                    
                                    Stack: [0x00000000024a0000,0x00000000025a0000]
                                    
                                    ---------------  P R O C E S S  ---------------
                                    
                                    Java Threads: ( => current thread )
                                      0x000000000720ac00 JavaThread "Timer-0" [_thread_blocked, id=4524, stack(0x0000003ef0350000,0x0000003ef6750000)]
                                      0x0000000007331400 JavaThread "SwingWorker-pool-1-thread-1" [_thread_blocked, id=1976, stack(0x0000003ee3b50000,0x0000003ee9f50000)]
                                      0x000000000098b800 JavaThread "DestroyJavaVM" [_thread_blocked, id=4272, stack(0x00000000085c0000,0x000000000e9c0000)]
                                      0x0000000007217800 JavaThread "TimerQueue" daemon [_thread_blocked, id=2940, stack(0x0000003ee9f50000,0x0000003ef0350000)]
                                      0x00000000070b8400 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=4716, stack(0x0000003edd750000,0x0000003ee3b50000)]
                                      0x0000000006ec8800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=3348, stack(0x000000006dc40000,0x0000000074040000)]
                                      0x0000000006ec8000 JavaThread "AWT-Shutdown" [_thread_blocked, id=3244, stack(0x0000000067840000,0x000000006dc40000)]
                                      0x0000000006ec5800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=1872, stack(0x0000000061440000,0x0000000067840000)]
                                      0x00000000068ebc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1888, stack(0x000000005b040000,0x0000000061440000)]
                                      0x00000000068eb400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=3292, stack(0x0000000007700000,0x0000000007800000)]
                                      0x0000000006cfec00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=840, stack(0x0000000007600000,0x0000000007700000)]
                                      0x0000000006cfe800 JavaThread "Attach Listener" daemon [_thread_blocked, id=2892, stack(0x0000000054c40000,0x000000005b040000)]
                                      0x0000000006cfe000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2504, stack(0x000000004e840000,0x0000000054c40000)]
                                      0x00000000068dc400 JavaThread "Finalizer" daemon [_thread_blocked, id=4028, stack(0x0000000048440000,0x000000004e840000)]
                                      0x0000000006e4a400 JavaThread "Reference Handler" daemon [_thread_blocked, id=4192, stack(0x0000000042040000,0x0000000048440000)]
                                    
                                    Other Threads:
                                      0x0000000006e45c00 VMThread [stack: 0x0000000007500000,0x0000000007600000] [id=2496]
                                      0x00000000068ef800 WatcherThread [stack: 0x0000000007800000,0x0000000007900000] [id=4616]
                                    
                                    =>0x000000000087f000 (exited) GCTaskThread [stack: 0x00000000024a0000,0x00000000025a0000] [id=2556]
                                    
                                    VM state:at safepoint (normal execution)
                                    
                                    VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
                                    [0x0000000000989ea0] UNKNOWN - owner thread: 0x0000000006e45c00
                                    [0x000000000098a440] UNKNOWN - owner thread: 0x000000000720ac00
                                    
                                    Heap
                                     PSYoungGen      total 75655040K, used 2532370K [0x0000002885400000, 0x0000003c72430000, 0x0000003c85400000)
                                      eden space 68154368K, 0% used [0x0000002885400000,0x0000002885400000,0x00000038c5100000)
                                      from space 7500672K, 33% used [0x00000038c5100000,0x000000395fa04a10,0x0000003a8ede0000)
                                      to   space 7292096K, 0% used [0x0000003ab5300000,0x0000003ab5300000,0x0000003c72430000)
                                     ParOldGen       total 15019264K, used 14337357K [0x0000000085400000, 0x0000000419f40000, 0x0000002885400000)
                                      object space 15019264K, 95% used [0x0000000085400000,0x00000003f0553460,0x0000000419f40000)
                                     PSPermGen       total 21504K, used 20713K [0x0000000080000000, 0x0000000081500000, 0x0000000085400000)
                                      object space 21504K, 96% used [0x0000000080000000,0x000000008143a650,0x0000000081500000)
                                    
                                    Dynamic libraries:
                                    0x0000000000400000 - 0x000000000042e000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\javaw.exe
                                    0x0000000077880000 - 0x0000000077a00000      C:\Windows\system32\ntdll.dll
                                    0x0000000077750000 - 0x000000007787b000      C:\Windows\system32\kernel32.dll
                                    0x000007feff4f0000 - 0x000007feff5f8000      C:\Windows\system32\ADVAPI32.dll
                                    0x000007feff770000 - 0x000007feff8af000      C:\Windows\system32\RPCRT4.dll
                                    0x0000000077680000 - 0x000000007774d000      C:\Windows\system32\USER32.dll
                                    0x000007fefec30000 - 0x000007fefec93000      C:\Windows\system32\GDI32.dll
                                    0x000007feffb50000 - 0x000007feffb7d000      C:\Windows\system32\IMM32.DLL
                                    0x000007fefee80000 - 0x000007fefef81000      C:\Windows\system32\MSCTF.dll
                                    0x000007feff9d0000 - 0x000007feffa6c000      C:\Windows\system32\msvcrt.dll
                                    0x000007feff9c0000 - 0x000007feff9cd000      C:\Windows\system32\LPK.DLL
                                    0x000007feff8b0000 - 0x000007feff94a000      C:\Windows\system32\USP10.dll
                                    0x0000000008000000 - 0x00000000085b7000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\server\jvm.dll
                                    0x000007fefbd40000 - 0x000007fefbd79000      C:\Windows\system32\WINMM.dll
                                    0x000007fefef90000 - 0x000007feff168000      C:\Windows\system32\ole32.dll
                                    0x000007feffa70000 - 0x000007feffb43000      C:\Windows\system32\OLEAUT32.dll
                                    0x000007fefbcf0000 - 0x000007fefbd3f000      C:\Windows\system32\OLEACC.dll
                                    0x0000000010000000 - 0x000000001000a000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\hpi.dll
                                    0x0000000077a00000 - 0x0000000077a09000      C:\Windows\system32\PSAPI.DLL
                                    0x00000000008d0000 - 0x00000000008de000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\verify.dll
                                    0x00000000008e0000 - 0x0000000000907000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\java.dll
                                    0x0000000000910000 - 0x0000000000922000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\zip.dll
                                    0x0000000007900000 - 0x0000000007ab5000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\awt.dll
                                    0x000007fefa130000 - 0x000007fefa188000      C:\Windows\system32\WINSPOOL.DRV
                                    0x000007fefdfd0000 - 0x000007fefec22000      C:\Windows\system32\SHELL32.dll
                                    0x000007feff210000 - 0x000007feff283000      C:\Windows\system32\SHLWAPI.dll
                                    0x000007fefc6c0000 - 0x000007fefc8b9000      C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6001.18000_none_152e7382f3bd50c6\comctl32.dll
                                    0x0000000007ec0000 - 0x0000000007f2b000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\fontmanager.dll
                                    0x00000000025b0000 - 0x00000000025c7000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\net.dll
                                    0x000007feff950000 - 0x000007feff994000      C:\Windows\system32\WS2_32.dll
                                    0x000007feff4e0000 - 0x000007feff4e7000      C:\Windows\system32\NSI.dll
                                    0x000007fefd360000 - 0x000007fefd3af000      C:\Windows\system32\mswsock.dll
                                    0x000007fefd400000 - 0x000007fefd407000      C:\Windows\System32\wship6.dll
                                    0x00000000025d0000 - 0x00000000025db000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\nio.dll
                                    0x000000000ed00000 - 0x000000000ed35000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\jpeg.dll
                                    0x000000000ed40000 - 0x000000000ed68000      C:\Program Files\Java\jdk1.6.0_10\jre\bin\dcpr.dll
                                    0x000007fefb640000 - 0x000007fefb6e0000      C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_40ba501d3c2b20ff\comctl32.dll
                                    
                                    VM Arguments:
                                    jvm_args: -Xmx240G -Xss100M -XX:+UseParallelOldGC 
                                    java_command: mc_square.Startup
                                    Launcher Type: SUN_STANDARD
                                    
                                    Environment Variables:
                                    PATH=C:\Program Files (x86)\Java\jre1.6.0_06\bin\client;C:\Program Files (x86)\Java\jre1.6.0_06\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem
                                    USERNAME=volker.kamin
                                    OS=Windows_NT
                                    PROCESSOR_IDENTIFIER=AMD64 Family 15 Model 65 Stepping 3, AuthenticAMD
                                    
                                    
                                    
                                    ---------------  S Y S T E M  ---------------
                                    
                                    OS: Windows Vista Build 6001 Service Pack 1
                                    
                                    CPU:total 16 (2 cores per cpu, 1 threads per core) family 15 model 65 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext
                                    
                                    Memory: 4k page, physical 267909756k(143199416k free), swap 402443376k(275397964k free)
                                    
                                    vm_info: Java HotSpot(TM) 64-Bit Server VM (11.0-b11) for windows-amd64 JRE (1.6.0_10-beta-b14), built on Mar 14 2008 13:53:18 by "java_re" with MS VC++ 8.0
                                    
                                    time: Wed May 07 13:04:27 2008
                                    elapsed time: 5985 seconds
                                    The Server OS is Windows Server 2008 Enterprise, don't know if that really is a Vista Build...
                                    I cannot reproduce the error, as the server is currently used for complex simulations.

                                    What I am wondering about is now:
                                    What is the real error? The overflow is of course an error, but did it cause the application to crash?
                                    1 2 Previous Next