1 Reply Latest reply: Nov 3, 2011 5:04 PM by jschellSomeoneStoleMyAlias RSS

    Fatal Error in JRE 6.0_24-b07

    897128
      I have found others facing a similar problem and yet mine is different and I'll be thankful if somebody could point to the possible source of this error and how to fix/diagnose it. I am running a Nutch crawler, writing to a Solr index on an EC2 instance with JRE version 6.0_24-b07 (though I have tried it with other versions as well). The crawler works fine on my local machine but very often the instance gets stuck and eventually creates an error file with contents similar to the following:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # Internal Error (safepoint.cpp:247), pid=16632, tid=140208175748864
      # guarantee(PageArmed == 0) failed: invariant
      #
      # JRE version: 6.0_24-b07
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64 compressed oops)
      # If you would like to submit a bug report, please visit:
      # http://java.sun.com/webapps/bugreport/crash.jsp
      #

      --------------- T H R E A D ---------------

      Current thread (0x0000000040e30000): VMThread [stack: 0x00007f84c2729000,0x00007f84c282a000] [id=16636]

      Stack: [0x00007f84c2729000,0x00007f84c282a000], sp=0x00007f84c2828a80, free space=1022k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x790f75]
      V [libjvm.so+0x324f06]
      V [libjvm.so+0x6b3ed6]
      V [libjvm.so+0x79f2a7]
      V [libjvm.so+0x79edbe]
      V [libjvm.so+0x64314f]

      VM_Operation (0x00007f84c072fb80): RevokeBias, mode: safepoint, requested by thread 0x0000000041d03000


      --------------- P R O C E S S ---------------

      Java Threads: ( => current thread )
      0x0000000041a41000 JavaThread "FetcherThread" daemon [_thread_new, id=18355, stack(0x0000000000000000,0x0000000000000000)]
      0x0000000041305800 JavaThread "FetcherThread" daemon [_thread_blocked, id=18354, stack(0x00007f84c032d000,0x00007f84c042e000)]
      0x00000000411f1000 JavaThread "FetcherThread" daemon [_thread_blocked, id=18353, stack(0x00007f84c052f000,0x00007f84c0630000)]
      0x00000000414b5000 JavaThread "FetcherThread" daemon [_thread_blocked, id=18352, stack(0x00007f84bbbfc000,0x00007f84bbcfd000)]
      0x0000000041645800 JavaThread "FetcherThread" daemon [_thread_blocked, id=18351, stack(0x00007f84bb3f4000,0x00007f84bb4f5000)]
      0x0000000041304800 JavaThread "FetcherThread" daemon [_thread_blocked, id=18350, stack(0x00007f84c002a000,0x00007f84c012b000)]
      0x0000000041d03000 JavaThread "QueueFeeder" daemon [_thread_blocked, id=18349, stack(0x00007f84c0630000,0x00007f84c0731000)]
      0x0000000041a1d000 JavaThread "SpillThread" daemon [_thread_blocked, id=18348, stack(0x00007f84c0d4c000,0x00007f84c0e4d000)]
      0x000000004118a800 JavaThread "communication thread" daemon [_thread_blocked, id=18347, stack(0x00007f84c0a40000,0x00007f84c0b41000)]
      0x00007f84bc4ea800 JavaThread "Thread-1105" [_thread_in_vm, id=18346, stack(0x00007f84c093f000,0x00007f84c0a40000)]
      0x000000004195d800 JavaThread "pool-2-thread-1" [_thread_blocked, id=16654, stack(0x00007f84c0c4b000,0x00007f84c0d4c000)]
      0x00007f84bc029800 JavaThread "ajp-bio-9009-AsyncTimeout" daemon [_thread_blocked, id=16652, stack(0x00007f84c0f45000,0x00007f84c1046000)]
      0x00007f84bc28b000 JavaThread "ajp-bio-9009-Acceptor-0" daemon [_thread_in_native, id=16651, stack(0x00007f84c1046000,0x00007f84c1147000)]
      0x00007f84bc005000 JavaThread "http-bio-9000-AsyncTimeout" daemon [_thread_blocked, id=16650, stack(0x00007f84c198d000,0x00007f84c1a8e000)]
      0x00007f84bc061000 JavaThread "http-bio-9000-Acceptor-0" daemon [_thread_in_native, id=16649, stack(0x00007f84c1147000,0x00007f84c1248000)]
      0x00007f84bc325000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=16648, stack(0x00007f84c1b55000,0x00007f84c1c56000)]
      0x00007f84bc328000 JavaThread "timerFactory" [_thread_blocked, id=16647, stack(0x00007f84c1248000,0x00007f84c1349000)]
      0x00000000410c2000 JavaThread "GC Daemon" daemon [_thread_blocked, id=16644, stack(0x00007f84c1dd8000,0x00007f84c1ed9000)]
      0x0000000040e64800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=16642, stack(0x00007f84c2123000,0x00007f84c2224000)]
      0x0000000040e5f800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=16641, stack(0x00007f84c2224000,0x00007f84c2325000)]
      0x0000000040e5c800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=16640, stack(0x00007f84c2325000,0x00007f84c2426000)]
      0x0000000040e5a800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=16639, stack(0x00007f84c2426000,0x00007f84c2527000)]
      0x0000000040e38800 JavaThread "Finalizer" daemon [_thread_blocked, id=16638, stack(0x00007f84c2527000,0x00007f84c2628000)]
      0x0000000040e36800 JavaThread "Reference Handler" daemon [_thread_blocked, id=16637, stack(0x00007f84c2628000,0x00007f84c2729000)]
      0x0000000040dd6800 JavaThread "main" [_thread_in_native, id=16633, stack(0x00007f84c7458000,0x00007f84c7559000)]

      Other Threads:
      =>0x0000000040e30000 VMThread [stack: 0x00007f84c2729000,0x00007f84c282a000] [id=16636]
      0x0000000040e6f000 WatcherThread [stack: 0x00007f84c2022000,0x00007f84c2123000] [id=16643]

      VM state:synchronizing (normal execution)

      VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
      [0x0000000040dd31a0] Safepoint_lock - owner thread: 0x0000000040e30000
      [0x0000000040dd3220] Threads_lock - owner thread: 0x0000000040e30000

      Heap
      PSYoungGen total 647744K, used 599873K [0x00000000d7f60000, 0x00000000ffee0000, 0x0000000100000000)
      eden space 641792K, 93% used [0x00000000d7f60000,0x00000000fc8e0700,0x00000000ff220000)
      from space 5952K, 5% used [0x00000000ff910000,0x00000000ff960000,0x00000000ffee0000)
      to space 6528K, 0% used [0x00000000ff220000,0x00000000ff220000,0x00000000ff880000)
      PSOldGen total 224704K, used 113284K [0x0000000087e00000, 0x0000000095970000, 0x00000000d7f60000)
      object space 224704K, 50% used [0x0000000087e00000,0x000000008eca10e8,0x0000000095970000)
      PSPermGen total 66176K, used 36389K [0x0000000082c00000, 0x0000000086ca0000, 0x0000000087e00000)
      object space 66176K, 54% used [0x0000000082c00000,0x0000000084f894b0,0x0000000086ca0000)

      Dynamic libraries:
      40000000-40009000 r-xp 00000000 08:01 25426 /usr/lib/jvm/java-6-sun-1.6.0.24/jre/bin/java
      40108000-4010a000 rwxp 00008000 08:01 25426 /usr/lib/jvm/java-6-sun-1.6.0.24/jre/bin/java
      40dcd000-43d79000 rwxp 00000000 00:00 0 [heap]
      82c00000-86ca0000 rwxp 00000000 00:00 0
      86ca0000-86db0000 ---p 00000000 00:00 0
      86db0000-87e00000 rwxp 00000000 00:00 0
      87e00000-95970000 rwxp 00000000 00:00 0
      95970000-d7f60000 ---p 00000000 00:00 0
      d7f60000-ffee0000 rwxp 00000000 00:00 0
      ffee0000-100000000 ---p 00000000 00:00 0
      7f84bb17b000-7f84bb1b8000 r-xs 0024f000 08:10 24445502 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/xmlbeans-2.3.0.jar
      7f84bb1b8000-7f84bb1bf000 r-xs 00049000 08:10 24445501 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/tika-parsers-0.9.jar
      7f84bb1bf000-7f84bb1c0000 r-xs 00015000 08:10 24445500 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/tagsoup-1.2.jar
      7f84bb1c0000-7f84bb1c2000 r-xs 00004000 08:10 24445499 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/slf4j-api-1.5.6.jar
      7f84bb1c2000-7f84bb1c6000 r-xs 0002f000 08:10 24445498 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/rome-0.9.jar
      7f84bb1c6000-7f84bb1d3000 r-xs 000c1000 08:10 24445497 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/poi-scratchpad-3.7.jar
      7f84bb1d3000-7f84bb252000 r-xs 0034a000 08:10 24445496 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/poi-ooxml-schemas-3.7.jar
      7f84bb252000-7f84bb259000 r-xs 00073000 08:10 24445495 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/poi-ooxml-3.7.jar
      7f84bb259000-7f84bb276000 r-xs 0017c000 08:10 24445494 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/poi-3.7.jar
      7f84bb276000-7f84bb28a000 r-xs 00339000 08:10 24445492 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/pdfbox-1.4.0.jar
      7f84bb28a000-7f84bb2bb000 r-xs 003f0000 08:10 24445490 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/netcdf-4.2-min.jar
      7f84bb2bb000-7f84bb2be000 r-xs 00014000 08:10 24445489 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/metadata-extractor-2.4.0-beta-1.jar
      7f84bb2be000-7f84bb2c0000 r-xs 0000b000 08:10 24445488 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/jempbox-1.4.0.jar
      7f84bb2c0000-7f84bb2c3000 r-xs 00023000 08:10 24445487 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/jdom-1.0.jar
      7f84bb2c3000-7f84bb2c5000 r-xs 00006000 08:10 24445486 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/geronimo-stax-api_1.0_spec-1.0.1.jar
      7f84bb2c5000-7f84bb2c9000 r-xs 00027000 08:10 24445485 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/fontbox-1.4.0.jar
      7f84bb2c9000-7f84bb2ce000 r-xs 00048000 08:10 24445484 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/dom4j-1.6.1.jar
      7f84bb2ce000-7f84bb2f3000 r-xs 00172000 08:10 24445478 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/bcprov-jdk15-1.45.jar
      7f84bb2f3000-7f84bb2f6000 ---p 00000000 00:00 0
      7f84bb2f6000-7f84bb3f4000 rwxp 00000000 00:00 0
      7f84bb3f4000-7f84bb3f7000 ---p 00000000 00:00 0
      7f84bb3f7000-7f84bb4f5000 rwxp 00000000 00:00 0
      7f84bb4f5000-7f84bb4f8000 ---p 00000000 00:00 0
      7f84bb4f8000-7f84bb5f6000 rwxp 00000000 00:00 0
      7f84bb5f6000-7f84bb5f9000 ---p 00000000 00:00 0
      7f84bb5f9000-7f84bb6f7000 rwxp 00000000 00:00 0
      7f84bb6f7000-7f84bb6fa000 ---p 00000000 00:00 0
      7f84bb6fa000-7f84bb7f8000 rwxp 00000000 00:00 0
      7f84bb7f8000-7f84bb7fb000 ---p 00000000 00:00 0
      7f84bb7fb000-7f84bb8f9000 rwxp 00000000 00:00 0
      7f84bb8f9000-7f84bb8fc000 ---p 00000000 00:00 0
      7f84bb8fc000-7f84bb9fa000 rwxp 00000000 00:00 0
      7f84bb9fa000-7f84bb9fd000 ---p 00000000 00:00 0
      7f84bb9fd000-7f84bbafb000 rwxp 00000000 00:00 0
      7f84bbafb000-7f84bbafe000 ---p 00000000 00:00 0
      7f84bbafe000-7f84bbbfc000 rwxp 00000000 00:00 0
      7f84bbbfc000-7f84bbbff000 ---p 00000000 00:00 0
      7f84bbbff000-7f84bbcfd000 rwxp 00000000 00:00 0
      7f84bbcfd000-7f84bbd00000 ---p 00000000 00:00 0
      7f84bbd00000-7f84bbdfe000 rwxp 00000000 00:00 0
      7f84bbdfe000-7f84bbe01000 ---p 00000000 00:00 0
      7f84bbe01000-7f84bbeff000 rwxp 00000000 00:00 0
      7f84bbeff000-7f84bbf02000 ---p 00000000 00:00 0
      7f84bbf02000-7f84bc000000 rwxp 00000000 00:00 0
      7f84bc000000-7f84bd5a9000 rwxp 00000000 00:00 0
      7f84bd5a9000-7f84c0000000 ---p 00000000 00:00 0
      7f84c0000000-7f84c0002000 r-xs 00001000 08:01 25328 /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/ext/dnsns.jar
      7f84c0002000-7f84c0004000 r-xs 0000d000 08:10 24445483 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/commons-logging-1.1.1.jar
      7f84c0004000-7f84c000a000 r-xs 00045000 08:10 24445482 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/commons-httpclient-3.1.jar
      7f84c000a000-7f84c000e000 r-xs 00024000 08:10 24445481 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-tika/commons-compress-1.1.jar
      7f84c000e000-7f84c0018000 r-xs 00088000 08:10 24445643 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/lib/zookeeper-3.3.1.jar
      7f84c0018000-7f84c0023000 r-xs 0005f000 08:10 24445642 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/lib/xstream-1.3.1.jar
      7f84c0023000-7f84c0025000 r-xs 00018000 08:10 24445451 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/lib-nekohtml/nekohtml-0.9.5.jar
      7f84c0025000-7f84c0026000 r-xs 00015000 08:10 24445469 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-html/tagsoup-1.2.jar
      7f84c0026000-7f84c0027000 r-xs 00000000 08:10 24445544 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/somethingparser/somethingparser.jar
      7f84c0027000-7f84c0028000 r-xs 00004000 08:10 24445467 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/parse-html/parse-html.jar
      7f84c0028000-7f84c0029000 r-xs 00003000 08:10 24445449 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/lib-http/lib-http.jar
      7f84c0029000-7f84c002a000 r-xs 00001000 08:10 24445511 /something1/packages/thecrawler/webapps/crawlers/WEB-INF/classes/conf/plugins/protocol-http/protocol-http.jar
      7f84c002a000-7f84c002d000 ---p 00000000 00:00 0
      7f84c002d000-7f84c012b000 rwxp 00000000 00:00 0
      7f84c012b000-7f84c012e000 ---p 00000000 00:00 0
      7f84c012e000-7f84c022c000 rwxp 00000000 00:00 0
      7f84c022c000-7f84c022f000 ---p 00000000 00:00 0
      7f84c022f000-7f84c032d000 rwxp 00000000 00:00 0
      7f84c032d000-7f84c0330000 ---p 00000000 00:00 0
      7f84c0330000-7f84c042e000 rwxp 00000000 00:00 0
      7f84c042e000-7f84c0431000 ---p 00000000 00:00 0
      .
      .
      .
      VM Arguments:
      vm_args: -Djava.util.logging.config.file=/something1/packages/thecrawler/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/something1/packages/apache-tomcat-7.0.20/endorsed -Dcatalina.base=/something1/packages/thecrawler -Dcatalina.home=/something1/packages/apache-tomcat-7.0.20 -Djava.io.tmpdir=/something1/packages/thecrawler/temp
      java_command: org.apache.catalina.startup.Bootstrap start
      Launcher Type: SUN_STANDARD

      Environment Variables:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
      USERNAME=root
      LD_LIBRARY_PATH=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/amd64
      SHELL=/bin/bash

      Signal Handlers:
      SIGSEGV: [libjvm.so+0x791b30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGBUS: [libjvm.so+0x791b30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGFPE: [libjvm.so+0x640ba0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGPIPE: [libjvm.so+0x640ba0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGXFSZ: [libjvm.so+0x640ba0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGILL: [libjvm.so+0x640ba0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGUSR2: [libjvm.so+0x643780], sa_mask[0]=0x00000000, sa_flags=0x10000004
      SIGHUP: [libjvm.so+0x643380], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGTERM: [libjvm.so+0x643380], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGQUIT: [libjvm.so+0x643380], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


      --------------- S Y S T E M ---------------

      OS:squeeze/sid

      uname:Linux 2.6.32-309-ec2 #18-Ubuntu SMP Mon Oct 18 21:00:50 UTC 2010 x86_64
      libc:glibc 2.11.1 NPTL 2.11.1
      rlimit: STACK 8192k, CORE 0k, NPROC infinity, NOFILE 1024, AS infinity
      load average:12.49 12.12 11.91

      /proc/meminfo:
      MemTotal: 7864548 kB
      MemFree: 562736 kB
      Buffers: 500660 kB
      Cached: 1560628 kB
      SwapCached: 0 kB
      Active: 4917260 kB
      Inactive: 1955200 kB
      Active(anon): 4198384 kB
      Inactive(anon): 612968 kB
      Active(file): 718876 kB
      Inactive(file): 1342232 kB
      Unevictable: 0 kB
      Mlocked: 0 kB
      SwapTotal: 0 kB
      SwapFree: 0 kB
      Dirty: 104 kB
      Writeback: 0 kB
      AnonPages: 4811024 kB
      Mapped: 666396 kB
      Shmem: 180 kB
      Slab: 157276 kB
      SReclaimable: 144948 kB
      SUnreclaim: 12328 kB
      KernelStack: 3032 kB
      PageTables: 0 kB
      NFS_Unstable: 0 kB
      Bounce: 0 kB
      WritebackTmp: 0 kB
      CommitLimit: 3932272 kB
      Committed_AS: 5022592 kB
      VmallocTotal: 34359738367 kB
      VmallocUsed: 6256 kB
      VmallocChunk: 34359731576 kB
      DirectMap4k: 7864320 kB
      DirectMap2M: 0 kB


      CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

      Memory: 4k page, physical 7864548k(562736k free), swap 0k(0k free)

      vm_info: Java HotSpot(TM) 64-Bit Server VM (19.1-b02) for linux-amd64 JRE (1.6.0_24-b07), built on Feb 2 2011 16:55:54 by "java_re" with gcc 3.2.2 (SuSE Linux)

      time: Thu Oct 13 08:54:34 2011
      elapsed time: 9913 seconds