9 Replies Latest reply: Mar 27, 2010 6:14 AM by 843798 RSS

    1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error

    843798
      Hi,

      I'm having a problem with the latest update of the JDK where all the binaries crash immediately with the following error:
      # ./java
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # java.lang.OutOfMemoryError: requested 64 bytes for char in /BUILD_AREA/jdk6_18/hotspot/src/share/vm/runtime/thread.cpp. Out of swap space?
      #
      #  Internal Error (allocation.inline.hpp:39), pid=26928, tid=1105066304
      #  Error: char in /BUILD_AREA/jdk6_18/hotspot/src/share/vm/runtime/thread.cpp
      #
      # JRE version: 6.0_18-b07
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
      # An error report file with more information is saved as:
      # /root/src/jre1.6.0_18/bin/hs_err_pid26928.log
      #
      # If you would like to submit a bug report, please visit:
      #   http://java.sun.com/webapps/bugreport/crash.jsp
      #
      Aborted
      This is on a 64-bit CentOS 5.4 machine:
      # uname -a
      Linux ... 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
      The error is actually different sometimes, but always something memory-related (and usually it's the one above).

      And it's definitely not out of swap space:
      # free -g
      total       used       free     shared    buffers     cached
      Mem:            62         52         10          0          0         43
      -/+ buffers/cache:          8         54
      Swap:           16          0         16
      Here's the log file: [hs_err_pid26928.log|http://dl.dropbox.com/u/624277/hs_err_pid26928.log|hs_err_pid26928.log]

      I tried both the JDK and the JRE distributions, same problem. The 32-bit JDK works just fine.

      Any ideas?

      Thanks
        • 1. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
          843798
          Hi DmitriB,

          Are you seeing this on multiple machines, with the same OS?
          Can you see this if you run something like 'java -version' ?

          Thanks,
          Roger
          • 2. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
            843798
            I don't actually have any other machines with the same environment (I have it running without problems on RHEL AS 4 and Debian 5). I can look into setting up a test image somewhere.

            Yes, it's the same error for any options/arguments. I also get what seems to be the same error, but reported differently from all other executables in the JDK:
            # ./javah
            Aborted
            
            # ./javac
            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # java.lang.OutOfMemoryError: requested 64 bytes for CHeapObj-new. Out of swap space?
            #
            #  Internal Error (allocation.inline.hpp:39), pid=27605, tid=1096378688
            #  Error: CHeapObj-new
            #
            # JRE version: 6.0_18-b07
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
            # An error report file with more information is saved as:
            # /root/src/jdk1.6.0_18/bin/hs_err_pid27605.log
            #
            # If you would like to submit a bug report, please visit:
            #   http://java.sun.com/webapps/bugreport/crash.jsp
            #
            Aborted
            
            # ./javadoc
            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # java.lang.OutOfMemoryError: pthread_getattr_np
            #
            #  Internal Error (os_linux_x86.cpp:681), pid=27784, tid=1116322112
            #  Error: pthread_getattr_np
            #
            # JRE version: 6.0_18-b07
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
            # An error report file with more information is saved as:
            # /root/src/jdk1.6.0_18/bin/hs_err_pid27784.log
            Aborted
            
            # ./jar
            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # java.lang.OutOfMemoryError: pthread_getattr_np
            #
            #  Internal Error (os_linux_x86.cpp:681), pid=27809, tid=1111468352
            #  Error: pthread_getattr_np
            #
            # JRE version: 6.0_18-b07
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
            # An error report file with more information is saved as:
            # /root/src/jdk1.6.0_18/bin/hs_err_pid27809.log
            *** glibc detected *** ./jar: double free or corruption (fasttop): 0x0000000041029960 ***
            Aborted
            Thanks,
            Dmitri
            • 3. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
              843798
              # ./java
              #
              # java.lang.OutOfMemoryError: requested 64 bytes for char in /BUILD_AREA/jdk6_18/hotspot/src/share/vm/runtime/thread.cpp. Out of swap space?
              #
              Aborted
              This is on a 64-bit CentOS 5.4 machine:
              # free -g
              total       used       free     shared    buffers     cached
              Mem:            62         52         10          0          0         43
              -/+ buffers/cache:          8         54
              Swap:           16          0         16
              With Java 5 or Java 6 on 64-bit linux and large amounts of memory the default max heap size should be 1GB
              and the default initial heap size with 64GB of ram should be 1GB.
              [http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#par_gc.ergonomics.default_size]

              Not convinced that always works;
              [http://forums.sun.com/thread.jspa?messageID=10871562#10871562]

              What output do you get from
              # ulimit -d -v
              • 4. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                843798
                I have the vm ulimit set to 16 gigs:
                # ulimit -d -v
                data seg size           (kbytes, -d) unlimited
                virtual memory          (kbytes, -v) 16777216
                It does seem to work if I remove the limit.

                With the limit, specifying a max heap size manually works, and it correctly reports trying to set the heap size too high, but there seems to be a range of sizes close to the maximum that throws the error:
                # ./java -Xmx1024m  -version
                java version "1.6.0_18"
                Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                
                # ./java -Xmx8m  -version
                java version "1.6.0_18"
                Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                
                # ./java -Xmx15360m  -version
                java version "1.6.0_18"
                Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                
                # ./java -Xmx16384m  -version
                Error occurred during initialization of VM
                Could not reserve enough space for object heap
                Could not create the Java virtual machine.
                
                # ./java -Xmx16100m  -version
                #
                # A fatal error has been detected by the Java Runtime Environment:
                #
                # java.lang.OutOfMemoryError: pthread_getattr_np
                #
                #  Internal Error (os_linux_x86.cpp:681), pid=10943, tid=1101519168
                #  Error: pthread_getattr_np
                #
                # JRE version: 6.0_18-b07
                # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
                # An error report file with more information is saved as:
                # /root/src/jdk1.6.0_18/bin/hs_err_pid10943.log
                #
                # If you would like to submit a bug report, please visit:
                #   http://java.sun.com/webapps/bugreport/crash.jsp
                #
                Aborted
                I was going to mention that 1.6.0u07 has been working fine for over a year, but it turns out that it just doesn't try to give nice errors for invalid sizes:
                # /opt/jdk1.6.0_07/bin/java -Xmx17g -version
                Error occurred during initialization of VM
                Could not reserve enough space for object heap
                *** glibc detected *** /opt/jdk1.6.0_07/bin/java: free(): invalid pointer: 0x0000000040f0df90 ***
                ======= Backtrace: =========
                /lib64/libc.so.6[0x3cfd6722ef]
                /lib64/libc.so.6(cfree+0x4b)[0x3cfd67273b]
                /opt/jdk1.6.0_07/jre/lib/amd64/server/libjvm.so[0x2b74ad54eb7e]
                /opt/jdk1.6.0_07/jre/lib/amd64/server/libjvm.so[0x2b74ad4f4b54]
                /opt/jdk1.6.0_07/jre/lib/amd64/server/libjvm.so[0x2b74ad4f6961]
                /opt/jdk1.6.0_07/jre/lib/amd64/server/libjvm.so[0x2b74ad4fa16c]
                /opt/jdk1.6.0_07/jre/lib/amd64/server/libjvm.so(JNI_CreateJavaVM+0x80)[0x2b74ad242f20]
                java[0x40003598]
                java(JavaMain+0x6e)[0x4000206e]
                /lib64/libpthread.so.0[0x3cfe206617]
                /lib64/libc.so.6(clone+0x6d)[0x3cfd6d3c2d]
                ======= Memory map: ========
                  [snipped]
                Aborted
                Is there a way to find out how much it actually tries to allocate by default?
                • 5. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                  843798
                  I can also replicate the "out of swap space?" error on the older RedHat system:
                  -bash-3.00$ uname -a
                  Linux ... 2.6.9-78.0.13.ELlargesmp #1 SMP Wed Jan 7 18:02:41 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
                  -bash-3.00$ ulimit -v
                  1048576
                  -bash-3.00$ for i in {1..10}; do c="./java -Xmx$((1024 - $i * 30))m -version"; echo $c; $c; done
                  
                  ...
                  
                  ./java -Xmx874m -version
                  Error occurred during initialization of VM
                  Could not reserve enough space for object heap
                  Could not create the Java virtual machine.
                  
                  ./java -Xmx844m -version
                  #
                  # A fatal error has been detected by the Java Runtime Environment:
                  #
                  # java.lang.OutOfMemoryError: requested 1048576 bytes for E in /BUILD_AREA/jdk6_18/hotspot/src/share/vm/utilities/taskqueue.hpp. Out of swap space?
                  #
                  #  Internal Error (allocation.inline.hpp:39), pid=1524, tid=1076017504
                  #  Error: E in /BUILD_AREA/jdk6_18/hotspot/src/share/vm/utilities/taskqueue.hpp
                  #
                  # JRE version: 6.0_18-b07
                  # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
                  # An error report file with more information is saved as:
                  # /home/bichkd/src/jre1.6.0_18/bin/hs_err_pid1524.log
                  #
                  # If you would like to submit a bug report, please visit:
                  #   http://java.sun.com/webapps/bugreport/crash.jsp
                  #
                  Aborted
                  
                  ./java -Xmx814m -version
                  java version "1.6.0_18"
                  Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                  Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                  
                  ...
                  But setting the same ulimit as the CentOS system does not cause problems with the default size on that machine (it's also 64GB).
                  • 6. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                    843798
                    DmitriB wrote:
                    I have the vm ulimit set to 16 gigs:
                    # ulimit -d -v
                    data seg size           (kbytes, -d) unlimited
                    virtual memory          (kbytes, -v) 16777216
                    It does seem to work if I remove the limit.
                    That is 16 MB not 16 GB.
                    • 7. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                      843798
                      Not unless it's flat-out lying about the units.

                      In any case:
                      # ./java -Xmx15g -version
                      java version "1.6.0_18"
                      Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                      Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                      
                      # ./java -Xmx17g -version
                      Error occurred during initialization of VM
                      Could not reserve enough space for object heap
                      Could not create the Java virtual machine.
                      • 8. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                        843798
                        DmitriB wrote:
                        Not unless it's flat-out lying about the units.
                        My bad. Should not post after bedtime.
                        • 9. Re: 1.6.0u18 on x86_64 Linux  crashes immediately with OutOfMemory error
                          843798
                          DmitriB wrote:
                          I have the vm ulimit set to 16 gigs:
                          You are using a vm? That could have some impact.
                          I have the vm ulimit set to 16 gigs:
                          # ulimit -d -v
                          data seg size           (kbytes, -d) unlimited
                          virtual memory          (kbytes, -v) 16777216
                          It does seem to work if I remove the limit.
                          On a 64-bit AMD box with just 4GB I get
                          # ulimit -v `echo 2^24|bc`
                          # ulimit -v
                          16777216
                          # java -Xmx15g -version
                          java version "1.6.0_18"
                          Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                          Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                          # java -Xmx16g -version
                          Error occurred during initialization of VM
                          Could not reserve enough space for object heap
                          Could not create the Java virtual machine.
                          # ulimit -v unlimited
                          # ulimit -v
                          unlimited
                          # java -Xmx84t -version
                          java version "1.6.0_18"
                          Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
                          Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
                          # java -Xmx85t -version
                          Error occurred during initialization of VM
                          Could not reserve space for ObjectStartArray