This discussion is archived
7 Replies Latest reply: Jul 14, 2011 12:25 PM by jschellSomeoneStoleMyAlias RSS

JNI causes 64 bit VM crash ( JRE 1.6-23 on RHEL 5.5 64 bit)

875120 Newbie
Currently Being Moderated
Hi Everybody,

we are experiencing JVM crash with JNI ( JBridge ) not able to dynamically link to a method ( atleast that is what we understand from the verbose JNI logs obtained using -verbose:jni, -Xcheck:jni VM parameters ).

Environment: OS: RHEL 5.5 ( 64 bit ) JRE: 1.6 update 23

1) Are there any known JNI issues with 64 bit JVM on RHEL 5.5 ?

2) Are there any other parameters ( other than the parameters used above ) which can help us pinpoint the problem ?

3) Are there any memory related parameters ( like -Xms/-Xmx ) specifically for JNI?

4) what are the tools that will be helpful to get additional stacktraces in c on RHEL to pinpoint the problem?

Below is the core dump log ( truncated ) we are getting ( could not find an option to attach the log file ).. Any suggestions are highly appreciated!!!


#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002aaa0033fd61, pid=19291, tid=1096268096
#
# JRE version: 6.0_23-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libEpiChannel.so+0x3cd2c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x00002aaab43e5000): JavaThread "ReadIMTables READER0" [_thread_in_native, id=20371, stack(0x000000004147b000,0x000000004157c000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00002aaa0033fd61

Registers:
RAX=0x00002aaa0033fd61, RBX=0x00000000bb8ccf08, RCX=0x0000000000000000, RDX=0x00000000000f7120
RSP=0x000000004157a718, RBP=0x000000004157a778, RSI=0x000000004157a7a0, RDI=0x00002aaab43e51c8
R8 =0x0000000046041580, R9 =0x00002b78982929d0, R10=0x00002aaaab17b9d9, R11=0x00002b7897fb43f0
R12=0x0000000000000000, R13=0x00000000bb8ccf08, R14=0x000000004157a7a0, R15=0x00002aaab43e5000
RIP=0x00002aaa0033fd61, EFL=0x0000000000010287, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
TRAPNO=0x000000000000000e

Register to memory mapping:

RAX=0x00002aaa0033fd61
0x00002aaa0033fd61 is pointing to unknown location

RBX=0x00000000bb8ccf08
{method}
- klass: {other class}

RCX=0x0000000000000000
0x0000000000000000 is pointing to unknown location

RDX=0x00000000000f7120
0x00000000000f7120 is pointing to unknown location

RSP=0x000000004157a718
0x000000004157a718 is pointing into the stack for thread: 0x00002aaab43e5000
"ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
java.lang.Thread.State: RUNNABLE

RBP=0x000000004157a778
0x000000004157a778 is pointing into the stack for thread: 0x00002aaab43e5000
"ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
java.lang.Thread.State: RUNNABLE

RSI=0x000000004157a7a0
0x000000004157a7a0 is pointing into the stack for thread: 0x00002aaab43e5000
"ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
java.lang.Thread.State: RUNNABLE

RDI=0x00002aaab43e51c8
0x00002aaab43e51c8 is pointing to unknown location

R8 =0x0000000046041580
0x0000000046041580 is pointing to unknown location

R9 =0x00002b78982929d0
0x00002b78982929d0: <offset 0x9759d0> in /u01/jre1.6.0_23/lib/amd64/server/libjvm.so at 0x00002b789791d000

R10=0x00002aaaab17b9d9
method entry point (kind = native) [0x00002aaaab17b740, 0x00002aaaab17c040] 2304 bytes

R11=0x00002b7897fb43f0
0x00002b7897fb43f0: <offset 0x6973f0> in /u01/jre1.6.0_23/lib/amd64/server/libjvm.so at 0x00002b789791d000

R12=0x0000000000000000
0x0000000000000000 is pointing to unknown location

R13=0x00000000bb8ccf08
{method}
- klass: {other class}

R14=0x000000004157a7a0
0x000000004157a7a0 is pointing into the stack for thread: 0x00002aaab43e5000
"ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
java.lang.Thread.State: RUNNABLE

R15=0x00002aaab43e5000
"ReadIMTables READER0" prio=10 tid=0x00002aaab43e5000 nid=0x4f93 runnable [0x000000004157a000]
java.lang.Thread.State: RUNNABLE


Top of Stack: (sp=0x000000004157a718)
0x000000004157a718: 00002aaaab17ba06 0000000045490930
0x000000004157a728: 0000000045490d08 00002aaab43e5000
0x000000004157a738: 000000004157a738 00000000bb8ccf08
0x000000004157a748: 000000004157a7a0 00000000bb9065f0
0x000000004157a758: 0000000000000000 00000000bb8ccf08
0x000000004157a768: 0000000000000000 000000004157a798
0x000000004157a778: 000000004157a7f0 00002aaaab16f9b3
0x000000004157a788: 0000000000000000 00002aaaab1788d7
0x000000004157a798: 00000000000f7120 00000000ee83d228
0x000000004157a7a8: 00000000ee83d228 000000004157a7b0
0x000000004157a7b8: 00000000bb8cd0af 000000004157a808
0x000000004157a7c8: 00000000bb9065f0 0000000000000000
0x000000004157a7d8: 00000000bb8cd0f0 000000004157a798
0x000000004157a7e8: 000000004157a800 000000004157a850
0x000000004157a7f8: 00002aaaab16fa82 00000000000f7120
0x000000004157a808: 00000000ee83d228 000000004157a810
0x000000004157a818: 00000000bb8cbf48 000000004157a880
0x000000004157a828: 00000000bb9066b8 0000000000000000
0x000000004157a838: 00000000bb8cbff8 000000004157a800
0x000000004157a848: 000000004157a860 000000004157a8e0
0x000000004157a858: 00002aaaab16fa82 00000000ee839408
0x000000004157a868: 00000000000001f4 00000000ee83cda8
0x000000004157a878: 0000000000000000 00000000ee83d228
0x000000004157a888: 00000000ee83d228 00002aaab43e5000
0x000000004157a898: 00000000ee83cda8 000000004157a890
0x000000004157a8a8: 00000000bb905eab 000000004157a8f8
0x000000004157a8b8: 00000000bb906448 0000000000000000
0x000000004157a8c8: 00000000bb905ed8 000000004157a860
0x000000004157a8d8: 000000004157a8f8 000000004157a940
0x000000004157a8e8: 00002aaaab16f929 0000000000000000
0x000000004157a8f8: 00000000ee83cda8 000000004157a900
0x000000004157a908: 00000000bb8bdfea 000000004157a9e0

Instructions: (pc=0x00002aaa0033fd61)
0x00002aaa0033fd51:
[error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb]

Stack: [0x000000004147b000,0x000000004157c000], sp=0x000000004157a718, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libEpiChannel.so+0x3cd2c]
j com.epiphany.etl.channel.ChannelBuffer.<init>(I)V+7
j com.epiphany.etl.channel.ChannelRowBatch.<init>(ILcom/epiphany/etl/channel/ChannelRowBatchPool;ILcom/epiphany/etl/channel/ChannelRowMetadata;)V+8
j com.epiphany.etl.channel.ChannelRowBatchPool.getBatch()Lcom/epiphany/etl/channel/ChannelRowBatch;+51
j com.epiphany.etl.channel.ChannelReader.run()V+266
j com.epiphany.shared.util.EpiThread$EpiRunnableRun.run()V+23
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x40a16f]
V [libjvm.so+0x622b89]
V [libjvm.so+0x409ad6]
V [libjvm.so+0x409b67]
V [libjvm.so+0x499907]
V [libjvm.so+0x7268c1]
V [libjvm.so+0x623e1f]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.epiphany.etl.channel.ChannelBuffer.createBuffer(I)J+0
j com.epiphany.etl.channel.ChannelBuffer.<init>(I)V+7
j com.epiphany.etl.channel.ChannelRowBatch.<init>(ILcom/epiphany/etl/channel/ChannelRowBatchPool;ILcom/epiphany/etl/channel/ChannelRowMetadata;)V+8
j com.epiphany.etl.channel.ChannelRowBatchPool.getBatch()Lcom/epiphany/etl/channel/ChannelRowBatch;+51
j com.epiphany.etl.channel.ChannelReader.run()V+266
j com.epiphany.shared.util.EpiThread$EpiRunnableRun.run()V+23
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00002aaab45f9800 JavaThread "ReadIMTables WRITER0" [_thread_blocked, id=20372, stack(0x000000004090c000,0x0000000040a0d000)]
=>0x00002aaab43e5000 JavaThread "ReadIMTables READER0" [_thread_in_native, id=20371, stack(0x000000004147b000,0x000000004157c000)]
0x0000000045913000 JavaThread "Temp Table Registration Renewer" [_thread_blocked, id=19419, stack(0x000000004137a000,0x000000004147b000)]
0x00002aaab4046800 JavaThread "Job Execution Thread" [_thread_blocked, id=19415, stack(0x000000004080b000,0x000000004090c000)]
0x00000000456a9800 JavaThread "Lock Manager PaceMaker" daemon [_thread_blocked, id=19411, stack(0x0000000040411000,0x0000000040512000)]
0x00002aaab4303000 JavaThread "Lock Manager Maintenance" daemon [_thread_blocked, id=19410, stack(0x00000000426a0000,0x00000000427a1000)]
0x00002aaab43e2000 JavaThread "DBLogWriter" [_thread_in_native, id=19370, stack(0x000000004179d000,0x000000004189e000)]
0x00002aaab42e2800 JavaThread "EpiChannel" [_thread_blocked, id=19338, stack(0x0000000041279000,0x000000004137a000)]
0x000000004534e800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=19335, stack(0x000000004249e000,0x000000004259f000)]
0x000000004534c800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=19334, stack(0x000000004239d000,0x000000004249e000)]
0x0000000045346800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=19333, stack(0x000000004229c000,0x000000004239d000)]
0x0000000045344800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19332, stack(0x000000004219b000,0x000000004229c000)]
0x0000000045322800 JavaThread "Finalizer" daemon [_thread_blocked, id=19331, stack(0x000000004209a000,0x000000004219b000)]
0x0000000045320800 JavaThread "Reference Handler" daemon [_thread_blocked, id=19330, stack(0x0000000041f99000,0x000000004209a000)]
0x00000000452c2800 JavaThread "main" [_thread_blocked, id=19326, stack(0x0000000041c96000,0x0000000041d97000)]

Other Threads:
0x000000004531c000 VMThread [stack: 0x0000000041e98000,0x0000000041f99000] [id=19329]
0x0000000045359000 WatcherThread [stack: 0x000000004259f000,0x00000000426a0000] [id=19336]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 336512K, used 70012K [0x00000000eaab0000, 0x0000000100000000, 0x0000000100000000)
eden space 323904K, 21% used [0x00000000eaab0000,0x00000000eee8f0b0,0x00000000fe700000)
from space 12608K, 4% used [0x00000000ff3b0000,0x00000000ff430000,0x0000000100000000)
to space 12800K, 0% used [0x00000000fe700000,0x00000000fe700000,0x00000000ff380000)
PSOldGen total 174784K, used 10583K [0x00000000c0000000, 0x00000000caab0000, 0x00000000eaab0000)
object space 174784K, 6% used [0x00000000c0000000,0x00000000c0a55f98,0x00000000caab0000)
PSPermGen total 21248K, used 11290K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
object space 21248K, 53% used [0x00000000bae00000,0x00000000bb9069e0,0x00000000bc2c0000)

VM Arguments:
jvm_args: -DGC_LOCALEDIR=/u03/Infor/Outbound_Marketing/7213_9046/locale -Depny.ap.root=/u03/Infor/Outbound_Marketing/7213_9046 -Xms256m -Xmx1024m
java_command: com.epiphany.etl.epichannel.EpiChannel -S inhywnpalapar1 -D naga_july4_meta1 -B PostgreSQL -U postgres -P Password@123 --LoggingDir ClientLogs -J InitialLoad
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/u01/jre1.6.0_23
CLASSPATH=/u03/Infor/Outbound_Marketing/7213_9046/lib/manifest.jar:/u03/Infor/Outbound_Marketing/7213_9046/classes:/u03/Infor/Outbound_Marketing/7213_9046/lib/AnalyticServices.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/catscan.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/bin/tools.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/metadata.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EpiChannel.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/ConfigService.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/core.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/db.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/logging.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/mail.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/activation.jar:/u03/Infor/Outbound_Marketing/7213_9046/Shared/lib/shared.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPbase.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPsqlserver.jar:/u03/Infor/Outbound_Marketing/7213_9046/lib/EPutil.jar:/usr/share/java/postgresql-8.4-702.jdbc3.jar
PATH=/u03/Infor/Outbound_Marketing/7213_9046/bin:/u01/jre1.6.0_23/bin:/u01/jre1.6.0_23/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
LD_LIBRARY_PATH=/u01/jre1.6.0_23/lib/amd64/server:/u01/jre1.6.0_23/lib/amd64:/u01/jre1.6.0_23/../lib/amd64:/u03/Infor/Outbound_Marketing/7213_9046/lib:/u03/Infor/Outbound_Marketing/7213_9046/odbc/lib:/u03/Infor/Outbound_Marketing/7213_9046/lib/stubs
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x76a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x76a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x621870], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x624450], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x624050], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


--------------- S Y S T E M ---------------

OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)

uname:Linux 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 16384, NOFILE 1024, AS infinity
load average:0.23 0.15 0.12

/proc/meminfo:
MemTotal: 2059516 kB
MemFree: 10264 kB
Buffers: 8012 kB
Cached: 312408 kB
SwapCached: 13608 kB
Active: 1692844 kB
Inactive: 234204 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 2059516 kB
LowFree: 10264 kB
SwapTotal: 4096532 kB
SwapFree: 4021232 kB
Dirty: 2476 kB
Writeback: 0 kB
AnonPages: 1605088 kB
Mapped: 37060 kB
Slab: 75808 kB
PageTables: 21244 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 5126288 kB
Committed_AS: 2902272 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 265016 kB
VmallocChunk: 34359472931 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB


CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

Memory: 4k page, physical 2059516k(10264k free), swap 4096532k(4021232k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (19.0-b09) for linux-amd64 JRE (1.6.0_23-b05), built on Nov 12 2010 14:12:21 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Tue Jul 12 15:14:00 2011
elapsed time: 19 seconds

Thanks,
Bangar Raju

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points