We have been facing a serious issue where the JVM in our servers keep crashing on an adhoc basis; To give a bit of context, this happened for the first time last year when we had JDK version 1.7_06 running on our servers and when this issue occur we took the decision to upgrade all the servers to 1.7_45. however in a few months (specifically in a months time) the same issue occurred again. In some instances we find the JVM in all the servers crashing, whilst in another instance we found the JVMs crash only in 2 servers. Hardware issues was ruled out as the servers themselves had varied memory configs and the latter instance of 2 servers only crashing didn't point to a common pattern here. A sample of the pid error log is provided below.
The application runs on CentOS 5.2 and Tomcat 6.0.35. One thing to note though. the code was not compiled into version 1.7._45. it was only the environment that was upgraded. could this be a problem?
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0x00002aaaab11e00c, pid=27138, tid=1570617664
# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libzip.so+0x1000c]*** glibc detected *** /xncpkgs/jdk1.7.0_45/bin/java: malloc(): memory corruption (fast):
[thread 1595881792 also had an error]
[thread 1545353536 also had an error]
[thread 1628514624 also had an error]
======= Backtrace: =========
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:21 14272741 /xncpkgs/jdk1.7.0_45/bin/java
00600000-00601000 rw-p 00000000 00:21 14272741 /xncpkgs/jdk1.7.0_45/bin/java
While researching on the web , a few things that I found was that
a.Java_java_util_zip_Deflater_deflateBytes is a known bug in jdk 1.7_45 and is associated with Tomcat 6.0. If so, is there a fix for this in JDK.1.7_51?
b. I've noticed the following de-optimization events as well in the pid logs
Deoptimization events (10 events):
Event: 1215.168 Thread 0x00000000136e1800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00002aaaaf9ffc30 method=java.util.AbstractSet.hashCode()I @ 17
Event: 1223.202 Thread 0x00002aaac0945000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00002aaaafb2fb3c method=com.sun.proxy.$Proxy46.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; @ 21
Event: 1260.951 Thread 0x00002aaac09b8000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00002aaab037b07c method=org.apache.velocity.runtime.parser.node.ASTEQNode.evaluate(Lorg/apache/velocity/context/InternalContextAdapter;)Z @ 73
Event: 1277.742 Thread 0x00002aaac07d6000 Uncommon trap: reason=null_check action=make_not_entrant pc=0x00002aaab0716ae8 method=org.apache.xerces.impl.XMLErrorReporter.reset(Lorg/apache/xerces/xni/parser/XMLComponentManager;)V @ 30
Event: 1282.303 Thread 0x00002aaab467c800 Uncommon trap: reason=unreached action=reinterpret pc=0x00002aaaaf7b1308 method=sun.reflect.GeneratedMethodAccessor303.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; @ 17
Event: 1370.843 Thread 0x00002aaac13e5000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00002aaaafa73128 method=org.springframework.transaction.support.DefaultTransactionStatus.isGlobalRollbackOnly()Z @ 4
Event: 1370.843 Thread 0x00002aaac13e5000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00002aaaaf960f3c method=org.hibernate.impl.SessionImpl.flush()V @ 56
Event: 1377.845 Thread 0x00000000136e1800 Uncommon trap: reason=array_check action=maybe_recompile pc=0x00002aaab006eb74 method=java.util.TimSort.binarySort([Ljava/lang/Object;IIILjava/util/Comparator;)V @ 183
Event: 1379.704 Thread 0x00002aaac0490800 Uncommon trap: reason=unreached action=reinterpret pc=0x00002aaab06e587c method=net.spy.memcached.MemcachedConnection.handleReads(Ljava/nio/channels/SelectionKey;Lnet/spy/memcached/MemcachedNode;)V @ 267
Event: 1382.314 Thread 0x00002aaad018f800 Uncommon trap: reason=unreached action=reinterpret pc=0x00002aaab06b586c method=com.singularity.ee.controller.api.dto.NameValuePair.convertNameValuePairsToMap([Lcom/singularity/ee/controller/api/dto/NameValuePair;)Ljava/util/Map; @ 14
Internal exceptions (10 events):
Event: 1382.323 Thread 0x00002aaac203d000 Threw 0x000000071a1402e8 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u45/229/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp:347
Event: 1382.476 Thread 0x00002aaac0151800 Threw 0x000000071c453da0 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u45/229/hotspot/src/share/vm/prims/jni.cpp:743
Event: 1382.578 Thread 0x00002aaac19a4800 Threw 0x000000071dfb2dd0 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u45/229/hotspot/src/share/vm/prims/jni.cpp:743
Does this mean that Java internally tries to recompile or delete truncate an application jar? is this why HUDSON is being called. Apologies for the long post, I would greatly appreciate the forums thought on this and especially the stack traces related to HUDSON and de-optimization.
thank you! this issue is very puzzling!