7 Replies Latest reply: Jul 14, 2011 2:25 PM by jschellSomeoneStoleMyAlias RSS

    JNI causes 64 bit VM crash ( JRE 1.6-23 on RHEL 5.5 64 bit)

    875120
      Hi Everybody,

      we are experiencing JVM crash with JNI ( JBridge ) not able to dynamically link to a method ( atleast that is what we understand from the verbose JNI logs obtained using -verbose:jni, -Xcheck:jni VM parameters ).

      Environment: OS: RHEL 5.5 ( 64 bit ) JRE: 1.6 update 23

      1) Are there any known JNI issues with 64 bit JVM on RHEL 5.5 ?

      2) Are there any other parameters ( other than the parameters used above ) which can help us pinpoint the problem ?

      3) Are there any memory related parameters ( like -Xms/-Xmx ) specifically for JNI?

      4) what are the tools that will be helpful to get additional stacktraces in c on RHEL to pinpoint the problem?

      Below is the core dump log ( truncated ) we are getting ( could not find an option to attach the log file ).. Any suggestions are highly appreciated!!!


      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x00002aaa0033fd61, pid=19291, tid=1096268096
      #
      # JRE version: 6.0_23-b05
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C [libEpiChannel.so+0x3cd2c]
      #
      # If you would like to submit a bug report, please visit:
      # http://java.sun.com/webapps/bugreport/crash.jsp
      # The crash happened outside the Java Virtual Machine in native code.
      # See problematic frame for where to report the bug.
      #

      --------------- T H R E A D ---------------

      Current thread (0x00002aaab43e5000): JavaThread "ReadIMTables READER0" [_thread_in_native, id=20371, stack(0x000000004147b000,0x000000004157c000)]

      siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00002aaa0033fd61

      Registers:
      RAX=0x00002aaa0033fd61, RBX=0x00000000bb8ccf08, RCX=0x0000000000000000, RDX=0x00000000000f7120
      RSP=0x000000004157a718, RBP=0x000000004157a778, RSI=0x000000004157a7a0, RDI=0x00002aaab43e51c8
      R8 =0x0000000046041580, R9 =0x00002b78982929d0, R10=0x00002aaaab17b9d9, R11=0x00002b7897fb43f0
      R12=0x0000000000000000, R13=0x00000000bb8ccf08, R14=0x000000004157a7a0, R15=0x00002aaab43e5000
      RIP=0x00002aaa0033fd61, EFL=0x0000000000010287, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
      TRAPNO=0x000000000000000e

      Register to memory mapping:

      RAX=0x00002aaa0033fd61
      0x00002aaa0033fd61 is pointing to unknown location

      RBX=0x00000000bb8ccf08
      {method}
      - klass: {other class}

      RCX=0x0000000000000000
      0x0000000000000000 is pointing to unknown location

      RDX=0x00000000000f7120
      0x00000000000f7120 is pointing to unknown location

      RSP=0x000000004157a718
      0x000000004157a718 is pointing into the stack for thread: 0x00002aaab43e5000
      "ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
      java.lang.Thread.State: RUNNABLE

      RBP=0x000000004157a778
      0x000000004157a778 is pointing into the stack for thread: 0x00002aaab43e5000
      "ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
      java.lang.Thread.State: RUNNABLE

      RSI=0x000000004157a7a0
      0x000000004157a7a0 is pointing into the stack for thread: 0x00002aaab43e5000
      "ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
      java.lang.Thread.State: RUNNABLE

      RDI=0x00002aaab43e51c8
      0x00002aaab43e51c8 is pointing to unknown location

      R8 =0x0000000046041580
      0x0000000046041580 is pointing to unknown location

      R9 =0x00002b78982929d0
      0x00002b78982929d0: <offset 0x9759d0> in /u01/jre1.6.0_23/lib/amd64/server/libjvm.so at 0x00002b789791d000

      R10=0x00002aaaab17b9d9
      method entry point (kind = native) [0x00002aaaab17b740, 0x00002aaaab17c040] 2304 bytes

      R11=0x00002b7897fb43f0
      0x00002b7897fb43f0: <offset 0x6973f0> in /u01/jre1.6.0_23/lib/amd64/server/libjvm.so at 0x00002b789791d000

      R12=0x0000000000000000
      0x0000000000000000 is pointing to unknown location

      R13=0x00000000bb8ccf08
      {method}
      - klass: {other class}

      R14=0x000000004157a7a0
      0x000000004157a7a0 is pointing into the stack for thread: 0x00002aaab43e5000
      "ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
      java.lang.Thread.State: RUNNABLE

      R15=0x00002aaab43e5000
      "ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
      java.lang.Thread.State: RUNNABLE


      Top of Stack: (sp=0x000000004157a718)
      0x000000004157a718: 00002aaaab17ba06 0000000045490930
      0x000000004157a728: 0000000045490d08 00002aaab43e5000
      0x000000004157a738: 000000004157a738 00000000bb8ccf08
      0x000000004157a748: 000000004157a7a0 00000000bb9065f0
      0x000000004157a758: 0000000000000000 00000000bb8ccf08
      0x000000004157a768: 0000000000000000 000000004157a798
      0x000000004157a778: 000000004157a7f0 00002aaaab16f9b3
      0x000000004157a788: 0000000000000000 00002aaaab1788d7
      0x000000004157a798: 00000000000f7120 00000000ee83d228
      0x000000004157a7a8: 00000000ee83d228 000000004157a7b0
      0x000000004157a7b8: 00000000bb8cd0af 000000004157a808
      0x000000004157a7c8: 00000000bb9065f0 0000000000000000
      0x000000004157a7d8: 00000000bb8cd0f0 000000004157a798
      0x000000004157a7e8: 000000004157a800 000000004157a850
      0x000000004157a7f8: 00002aaaab16fa82 00000000000f7120
      0x000000004157a808: 00000000ee83d228 000000004157a810
      0x000000004157a818: 00000000bb8cbf48 000000004157a880
      0x000000004157a828: 00000000bb9066b8 0000000000000000
      0x000000004157a838: 00000000bb8cbff8 000000004157a800
      0x000000004157a848: 000000004157a860 000000004157a8e0
      0x000000004157a858: 00002aaaab16fa82 00000000ee839408
      0x000000004157a868: 00000000000001f4 00000000ee83cda8
      0x000000004157a878: 0000000000000000 00000000ee83d228
      0x000000004157a888: 00000000ee83d228 00002aaab43e5000
      0x000000004157a898: 00000000ee83cda8 000000004157a890
      0x000000004157a8a8: 00000000bb905eab 000000004157a8f8
      0x000000004157a8b8: 00000000bb906448 0000000000000000
      0x000000004157a8c8: 00000000bb905ed8 000000004157a860
      0x000000004157a8d8: 000000004157a8f8 000000004157a940
      0x000000004157a8e8: 00002aaaab16f929 0000000000000000
      0x000000004157a8f8: 00000000ee83cda8 000000004157a900
      0x000000004157a908: 00000000bb8bdfea 000000004157a9e0

      Instructions: (pc=0x00002aaa0033fd61)
      0x00002aaa0033fd51:
      [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb]

      Stack: [0x000000004147b000,0x000000004157c000], sp=0x000000004157a718, free space=1021k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C [libEpiChannel.so+0x3cd2c]
      j com.epiphany.etl.channel.ChannelBuffer.<init>(I)V+7
      j com.epiphany.etl.channel.ChannelRowBatch.<init>(ILcom/epiphany/etl/channel/ChannelRowBatchPool;ILcom/epiphany/etl/channel/ChannelRowMetadata;)V+8
      j com.epiphany.etl.channel.ChannelRowBatchPool.getBatch()Lcom/epiphany/etl/channel/ChannelRowBatch;+51
      j com.epiphany.etl.channel.ChannelReader.run()V+266
      j com.epiphany.shared.util.EpiThread$EpiRunnableRun.run()V+23
      j java.lang.Thread.run()V+11
      v ~StubRoutines::call_stub
      V [libjvm.so+0x40a16f]
      V [libjvm.so+0x622b89]
      V [libjvm.so+0x409ad6]
      V [libjvm.so+0x409b67]
      V [libjvm.so+0x499907]
      V [libjvm.so+0x7268c1]
      V [libjvm.so+0x623e1f]

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      j com.epiphany.etl.channel.ChannelBuffer.createBuffer(I)J+0
      j com.epiphany.etl.channel.ChannelBuffer.<init>(I)V+7
      j com.epiphany.etl.channel.ChannelRowBatch.<init>(ILcom/epiphany/etl/channel/ChannelRowBatchPool;ILcom/epiphany/etl/channel/ChannelRowMetadata;)V+8
      j com.epiphany.etl.channel.ChannelRowBatchPool.getBatch()Lcom/epiphany/etl/channel/ChannelRowBatch;+51
      j com.epiphany.etl.channel.ChannelReader.run()V+266
      j com.epiphany.shared.util.EpiThread$EpiRunnableRun.run()V+23
      j java.lang.Thread.run()V+11
      v ~StubRoutines::call_stub

      --------------- P R O C E S S ---------------

      Java Threads: ( => current thread )
      0x00002aaab45f9800 JavaThread "ReadIMTables WRITER0" [_thread_blocked, id=20372, stack(0x000000004090c000,0x0000000040a0d000)]
      =>0x00002aaab43e5000 JavaThread "ReadIMTables READER0" [_thread_in_native, id=20371, stack(0x000000004147b000,0x000000004157c000)]
      0x0000000045913000 JavaThread "Temp Table Registration Renewer" [_thread_blocked, id=19419, stack(0x000000004137a000,0x000000004147b000)]
      0x00002aaab4046800 JavaThread "Job Execution Thread" [_thread_blocked, id=19415, stack(0x000000004080b000,0x000000004090c000)]
      0x00000000456a9800 JavaThread "Lock Manager PaceMaker" daemon [_thread_blocked, id=19411, stack(0x0000000040411000,0x0000000040512000)]
      0x00002aaab4303000 JavaThread "Lock Manager Maintenance" daemon [_thread_blocked, id=19410, stack(0x00000000426a0000,0x00000000427a1000)]
      0x00002aaab43e2000 JavaThread "DBLogWriter" [_thread_in_native, id=19370, stack(0x000000004179d000,0x000000004189e000)]
      0x00002aaab42e2800 JavaThread "EpiChannel" [_thread_blocked, id=19338, stack(0x0000000041279000,0x000000004137a000)]
      0x000000004534e800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=19335, stack(0x000000004249e000,0x000000004259f000)]
      0x000000004534c800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=19334, stack(0x000000004239d000,0x000000004249e000)]
      0x0000000045346800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=19333, stack(0x000000004229c000,0x000000004239d000)]
      0x0000000045344800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19332, stack(0x000000004219b000,0x000000004229c000)]
      0x0000000045322800 JavaThread "Finalizer" daemon [_thread_blocked, id=19331, stack(0x000000004209a000,0x000000004219b000)]
      0x0000000045320800 JavaThread "Reference Handler" daemon [_thread_blocked, id=19330, stack(0x0000000041f99000,0x000000004209a000)]
      0x00000000452c2800 JavaThread "main" [_thread_blocked, id=19326, stack(0x0000000041c96000,0x0000000041d97000)]

      Other Threads:
      0x000000004531c000 VMThread [stack: 0x0000000041e98000,0x0000000041f99000] [id=19329]
      0x0000000045359000 WatcherThread [stack: 0x000000004259f000,0x00000000426a0000] [id=19336]

      VM state:not at safepoint (normal execution)

      VM Mutex/Monitor currently owned by a thread: None

      Heap
      PSYoungGen total 336512K, used 70012K [0x00000000eaab0000, 0x0000000100000000, 0x0000000100000000)
      eden space 323904K, 21% used [0x00000000eaab0000,0x00000000eee8f0b0,0x00000000fe700000)
      from space 12608K, 4% used [0x00000000ff3b0000,0x00000000ff430000,0x0000000100000000)
      to space 12800K, 0% used [0x00000000fe700000,0x00000000fe700000,0x00000000ff380000)
      PSOldGen total 174784K, used 10583K [0x00000000c0000000, 0x00000000caab0000, 0x00000000eaab0000)
      object space 174784K, 6% used [0x00000000c0000000,0x00000000c0a55f98,0x00000000caab0000)
      PSPermGen total 21248K, used 11290K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 53% used [0x00000000bae00000,0x00000000bb9069e0,0x00000000bc2c0000)

      VM Arguments:
      jvm_args: -DGC_LOCALEDIR=/u03/Infor/Outbound_Marketing/7213_9046/locale -Depny.ap.root=/u03/Infor/Outbound_Marketing/7213_9046 -Xms256m -Xmx1024m
      java_command: com.epiphany.etl.epichannel.EpiChannel -S inhywnpalapar1 -D naga_july4_meta1 -B PostgreSQL -U postgres -P Password@123 --LoggingDir ClientLogs -J InitialLoad
      Launcher Type: SUN_STANDARD

      Environment Variables:
      JAVA_HOME=/u01/jre1.6.0_23
      CLASSPATH=/u03/Infor/Outbound_Marketing/7213_9046/lib/manifest.jar:/u03/Infor/Outbound_Marketing/7213_9046/classes:/u03/Infor/Outbound_Marketing/7213_9046/lib/AnalyticServices.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/catscan.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/bin/tools.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/metadata.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EpiChannel.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/ConfigService.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/core.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/db.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/logging.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/mail.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/activation.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/shared.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPbase.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPsqlserver.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPutil.jar:/usr/share/java/postgresql-8.4-702.jdbc3.jar
      PATH=/u03/Infor/Outbound_Marketing/7213_9046/bin:/u01/jre1.6.0_23/bin:/u01/jre1.6.0_23/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
      LD_LIBRARY_PATH=/u01/jre1.6.0_23/lib/amd64/server:/u01/jre1.6.0_23/lib/amd64:/u01/jre1.6.0_23/../lib/amd64:/u03/Infor/Outbound_Marketing/7213_9046/lib:/u03/Infor/Outbound_Marketing/7213_9046/odbc/lib:/u03/Infor/Outbound_Marketing/7213_9046/lib/stubs
      SHELL=/bin/bash

      Signal Handlers:
      SIGSEGV: [libjvm.so+0x76a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGBUS: [libjvm.so+0x76a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGFPE: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGPIPE: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGXFSZ: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGILL: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGUSR2: [libjvm.so+0x624450], sa_mask[0]=0x00000000, sa_flags=0x10000004
      SIGHUP: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGINT: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGTERM: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
      SIGQUIT: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


      --------------- S Y S T E M ---------------

      OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)

      uname:Linux 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64
      libc:glibc 2.5 NPTL 2.5
      rlimit: STACK 10240k, CORE 0k, NPROC 16384, NOFILE 1024, AS infinity
      load average:0.23 0.15 0.12

      /proc/meminfo:
      MemTotal: 2059516 kB
      MemFree: 10264 kB
      Buffers: 8012 kB
      Cached: 312408 kB
      SwapCached: 13608 kB
      Active: 1692844 kB
      Inactive: 234204 kB
      HighTotal: 0 kB
      HighFree: 0 kB
      LowTotal: 2059516 kB
      LowFree: 10264 kB
      SwapTotal: 4096532 kB
      SwapFree: 4021232 kB
      Dirty: 2476 kB
      Writeback: 0 kB
      AnonPages: 1605088 kB
      Mapped: 37060 kB
      Slab: 75808 kB
      PageTables: 21244 kB
      NFS_Unstable: 0 kB
      Bounce: 0 kB
      CommitLimit: 5126288 kB
      Committed_AS: 2902272 kB
      VmallocTotal: 34359738367 kB
      VmallocUsed: 265016 kB
      VmallocChunk: 34359472931 kB
      HugePages_Total: 0
      HugePages_Free: 0
      HugePages_Rsvd: 0
      Hugepagesize: 2048 kB


      CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

      Memory: 4k page, physical 2059516k(10264k free), swap 4096532k(4021232k free)

      vm_info: Java HotSpot(TM) 64-Bit Server VM (19.0-b09) for linux-amd64 JRE (1.6.0_23-b05), built on Nov 12 2010 14:12:21 by "java_re" with gcc 3.2.2 (SuSE Linux)

      time: Tue Jul 12 15:14:00 2011
      elapsed time: 19 seconds

      Thanks,
      Bangar Raju