2 Replies Latest reply: Dec 17, 2010 8:45 AM by user8958419 RSS

    R28.0 Illegal memory access. [54]

    806769
      Hi All,

      Hoping someone can point me in the right direction on the below crash. It isn't happening that frequently. I have 17 production instances running and have been getting about one crash per week. Previous crashes were on R27 and the dump pointed clearly to a jrockit method. I upgraded to jrockit R28 and after just over 48 hours one instance crashed with the below dump. I reviewed the troubleshooting documentation and nothing there is jumping out at me as applying to this. I see in the stack trace that it is around JasperReports & WebLogic XML processing but since it is a memory access issue wanted to start here first. One other tidbit, I did a deploy of applications about 40 minutes before this happened, but my other instances didn't experience a problem.

      Thanks!


      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 2 days, 02:28:08 on Tue Oct 19 14:42:06 2010

      **************************************************************************
      * If you see this dump, please go to *
      * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
      * for troubleshooting information. *
      **************************************************************************

      Additional information is available in:
      /opt/wls/user_projects/domains/PROD/jrockit.4776.dump
      No snapshot file (core dump) will be created because core dumps have been
      disabled. To enable core dumping, try "ulimit -c unlimited"
      before starting JRockit again.

      Error Message: Illegal memory access. [54]
      Signal info : si_signo=11, si_code=1 si_addr=0x10000
      Version : Oracle JRockit(R) R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-ia32
      CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 Core Intel64
      Number CPUs : 8
      Tot Phys Mem : 16805838848 (16027 MB)
      OS version : Red Hat Enterprise Linux ES release 4 (Nahant Update 6)
      Linux version 2.6.9-67.ELsmp (brewbuilder@hs20-bc1-5.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Wed Nov 7 13:56:44 EST 2007 (x86_64)
      Thread System: Linux NPTL
      LibC release : 2.3.4-stable
      Java locking : Lazy unlocking enabled (class banning) (transfer banning)
      State : JVM is running
      Command Line : -Xms3g -Xmx3g -Xverify:none -da -Dplatform.home=/opt/wls/wlserver_10.3 -Dwls.home=/opt/wls/wlserver_10.3/server -Dweblogic.home=/opt/wls/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/opt/wls/patch_wls1030/profiles/default/sysext_manifest_classpath:/opt/wls/patch_cie660/profiles/default/sysext_manifest_classpath -Xverbose:gcpause -Xmanagement:port=7009,ssl=false,authenticate=false -Dnewops_userId=AutomationMachine -Dnewops_password=DHLGlobalMa1l! -Dweblogic.Name=AdminServer -Djava.security.policy=/opt/wls/wlserver_10.3/server/lib/weblogic.policy -Dsun.java.launcher=SUN_STANDARD weblogic.Server
      Repository : /tmp/2010_10_17_12_14_00_4776
      java.home : /opt/wls/jrmc-4.0.1-1.6.0/jre
      j.class.path : :/opt/wls/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/wls/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/wls/jrmc-4.0.1-1.6.0/lib/tools.jar:/opt/wls/wlserver_10.3/server/lib/weblogic_sp.jar:/opt/wls/wlserver_10.3/server/lib/weblogic.jar:/opt/wls/modules/features/weblogic.server.modules_10.3.0.0.jar:/opt/wls/wlserver_10.3/server/lib/webservices.jar:/opt/wls/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/opt/wls/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar::/opt/wls/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/opt/wls/wlserver_10.3/server/lib/xqrl.jar:::/opt/wls/wlserver_10.3/server/lib/dms.jar:/opt/wls/wlserver_10.3/server/lib/ojdbc6.jar
      j.lib.path : /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/jrockit:/opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386:/opt/wls/jrmc-4.0.1-1.6.0/jre/../lib/i386::/opt/wls/wlserver_10.3/server/native/linux/i686:/opt/wls/wlserver_10.3/server/native/linux/i686/oci920_8
      JAVA_HOME : /opt/wls/jrmc-4.0.1-1.6.0
      JAVAOPTIONS: <not set>
      LD_LIBRARY_PATH: /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/jrockit:/opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386:/opt/wls/jrmc-4.0.1-1.6.0/jre/../lib/i386::/opt/wls/wlserver_10.3/server/native/linux/i686:/opt/wls/wlserver_10.3/server/native/linux/i686/oci920_8
      LD_ASSUME_KERNEL: <not set>
      LD_PRELOAD : <not set>
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory : 0 OutOfMemoryErrors have occured
      C Heap : Good; no memory allocations have failed
      GC Strategy : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
      GC Status : OC is not running. Last finished OC was OC#145.
      : YC is not running. Last finished YC was YC#759.
      YC Promotion : Last YC successfully promoted all objects
      YC History : Ran 0 YCs before OC#141.
      : Ran 8 YCs before OC#142.
      : Ran 5 YCs before OC#143.
      : Ran 4 YCs before OC#144.
      : Ran 4 YCs before OC#145.
      : Ran 0 YCs since last OC.
      Heap : 0x3668f000 - 0xf668f000 (Size: 3072 MB)
      Compaction : (no compaction area)
      NurseryList : 0x5dc00fa0 - 0xae1f8ff0
      KeepArea : 0x9c134678 - 0xae1f8ff0
      KA Markers : [ 0x86e36e70,  0x9c134678 , 0xae1f8ff0 ]
      Forbidden A : (none)
      Previous KA : 0x83eff7f0 - 0x994f6910
      Previous FA : (none)
      CompRefs : References are 32-bit.

      Registers (from ThreadContext: 0x12a562c0:
      eax = 00000000 ecx = 00000000 edx = 00000000 ebx = 6ba323e0
      esp = 12a565ac ebp = 3de95ec8 esi = 00000000 edi = 00000000
      es = 0000002b cs = 0bb10023 ss = 8107002b ds = 0000002b
      fs = 00000003 gs = 0000005b
      eip = 00010000 eflags = 00000246

      Loaded modules:
      (* denotes the module where the exception occured)
      08048000-08057173 /opt/wls/jrmc-4.0.1-1.6.0/bin/java
      ffffe000-ffffe7bf /opt/wls/jrmc-4.0.1-1.6.0/bin/java
      00a1a000-00a1b967 /lib/libdl.so.2
      00b1e000-00b2b85b /lib/tls/libpthread.so.0
      008ec000-00a11578 /lib/tls/libc.so.6
      008cd000-008e239b /lib/ld-linux.so.2
      f7ca1000-f7f55eb7 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/jrockit/libjvm.so
      f7c85000-f7c9ec03 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libosal.so
      f7c79000-f7c8349f /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libutil.so
      00a20000-00a40c8f /lib/tls/libm.so.6
      f7c33000-f7c3aa76 /lib/tls/librt.so.1
      f70f5000-f70fda57 /lib/libnss_files.so.2
      f705b000-f7066f47 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libjfr.so
      2fee0000-2feea9bb /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libverify.so
      2febb000-2fedde37 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libjava.so
      00a45000-00a5766f /lib/libnsl.so.1
      2fe54000-2fe59e50 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/native_threads/libhpi.so
      2fa0f000-2fa1d2e4 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libzip.so
      2f545000-2f54a666 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libmanagement.so
      2f3a1000-2f3b3ed8 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libnet.so
      2ee9b000-2eea1424 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libnio.so
      2e184000-2e185de4 /opt/wls/wlserver_10.3/server/native/linux/i686/libwlfileio2.so
      2dedc000-2dedcd53 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/liborii.so
      288ac000-288af313 /lib/libnss_dns.so.2
      00b62000-00b70fef /lib/libresolv.so.2
      1ebfe000-1ebfe6ad /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/librmi.so
      190ea000-1916dd26 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libawt.so
      190e4000-190e8228 /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/headless/libmawt.so
      18ca6000-18d24aef /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libfontmanager.so
      180f7000-18124baf /opt/wls/jrmc-4.0.1-1.6.0/jre/lib/i386/libjpeg.so
      15b19000-15b1b27f /tmp/usps4cb-769724432549500680.so
      152db000-152dd27f /tmp/usps4cb-4034220135931723258.so

      Stack:
      (* marks the word pointed to by the stack pointer)
      12a565ac: 123ac48d* 3de95ec8 00000001 00000001 81072be8 81073158
      12a565c4: 6ba32270 6ba31ef0 3de96e60 6ba323c0 00000000 6ba32270
      12a565dc: 2d1f7673 6b9e2fa0 00000000 6ba32270 2d1f762f 6b9e2fa0
      12a565f4: 6ba31ef0 6ba266a0 00000000 6b9899e0 6ba266a0 6ba31ef0

      Code:
      (* marks the word pointed to by the instruction pointer)
      0000ffd0: 00000000 00000000 00000000 00000000 00000000 00000000
      0000ffe8: 00000000 00000000 00000000 00000000 00000000 00000000
      00010000: Unreadable
      00010018: Unreadable


      Last optimized methods:
      #5617 com/dpwn/newops/encoding/server/service/adapter/MailItemsConverter.convertBo(Lcom/dpwn/newops/encoding/server/bo/encoding/MailItemBo;Lcom/dpwn/newops/routing/serviceapi/dto/RoutingMatrixInputDto;)Lcom/dpwn/newops/routing/serviceapi/dto/RoutingMatrixInputDto; 181683.633-181683.794 0x84a580-0x84c3f2 160.68 ms 1984KB
      #5618 com/dpwn/newops/framework/server/converter/AbstractConverter.convertBigDecimal(Ljava/math/BigDecimal;)Lcom/dpwn/newops/framework/shared/datatype/Quantity; 181683.794-181683.864 0x65bcc0-0x65c83e 70.07 ms 1152KB
      #5619 org/hibernate/impl/SessionFactoryImpl.getCollectionPersister(Ljava/lang/String;)Lorg/hibernate/persister/collection/CollectionPersister; 181683.936-181683.938 0x84c400-0x84c4ad 1.93 ms 256KB
      #5620 sun/reflect/GeneratedMethodAccessor7984.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;  181684.003-181684.006 0x84c4c0-0x84c635 2.98 ms 256KB
      #5621 org/hibernate/type/NullableType.isEqual(Ljava/lang/Object;Ljava/lang/Object;Lorg/hibernate/EntityMode;)Z  181684.026-181684.027 0x84c640-0x84c653 0.67 ms 128KB
      #5622 org/hibernate/event/def/DefaultLoadEventListener.onLoad(Lorg/hibernate/event/LoadEvent;Lorg/hibernate/event/LoadEventListener$LoadType;)V  181684.157-181684.520 0x1956f000-0x19570f6e 363.12 ms 2752KB
      #5623 org/hibernate/event/def/DefaultPersistEventListener.onPersist(Lorg/hibernate/event/PersistEvent;Ljava/util/Map;)V  181684.521-181684.744 0x19570f80-0x195726fb 223.62 ms 2240KB
      #5624 org/hibernate/util/FastHashMap.get(Ljava/lang/Object;)Ljava/lang/Object;  181684.744-181684.745 0x19572700-0x19572713 0.67 ms 256KB
      #5625 org/hibernate/type/TimestampType.isEqual(Ljava/lang/Object;Ljava/lang/Object;)Z  181684.745-181684.751 0x19572720-0x195728ce 6.06 ms 256KB
      #5626 org/hibernate/engine/StatefulPersistenceContext.getEntry(Ljava/lang/Object;)Lorg/hibernate/engine/EntityEntry;  181684.751-181684.752 0x195728e0-0x1957291a 0.73 ms 128KB
      #5627 org/hibernate/event/def/AbstractFlushingEventListener.flushEntities(Lorg/hibernate/event/FlushEvent;)V  181684.752-181684.771 0x19572920-0x19572ddb 18.96 ms 448KB
      #5628 org/hibernate/jdbc/ConnectionManager.closeConnection()V  181685.091-181685.175 0x19572de0-0x195737be 83.51 ms 1024KB
      #5629 org/hibernate/util/IdentityMap.entryList()Ljava/util/List;  181685.473-181686.069 0x195737c0-0x1957589c 596.23 ms 3584KB
      * #5630 org/hibernate/event/def/AbstractFlushingEventListener.flushEverythingToExecutions(Lorg/hibernate/event/FlushEvent;)V  181686.069-181686.716 0x195758a0-0x195782bd 646.38 ms 4416KB
      #5615 org/hibernate/jdbc/ConnectionManager.closeConnection()V  181683.371-181683.452 0x8486a0-0x84907e 80.94 ms 960KB
      #5616 org/springframework/orm/jpa/ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;  181683.452-181683.633 0x849080-0x84a57d 181.09 ms 1920KB

      Thread:
      "[ACTIVE] ExecuteThread: '9' for" id=1114 idx=0x110 tid=27106 lastJavaFrame=(nil)
      Stack 0: start=0x12a36000, end=0x12a58000, guards=0x12a3b000 (ok), forbidden=0x12a39000

      Thread Stack Trace:
      -- Java stack --
      at weblogic/xml/registry/XMLRegistry.isHandleEntityInvalidation(XMLRegistry.java:1289)[optimized]
      at weblogic/xml/registry/RegistryEntityResolver.isHandleEntityInvalidation(RegistryEntityResolver.java:137)
      at weblogic/xml/jaxp/RegistryXMLReader.parse(RegistryXMLReader.java:159)
      at org/apache/commons/digester/Digester.parse(Digester.java:1745)
      at net/sf/jasperreports/engine/xml/JRXmlLoader.loadXML(JRXmlLoader.java:239)
      at net/sf/jasperreports/engine/xml/JRXmlLoader.loadXML(JRXmlLoader.java:226)
      at net/sf/jasperreports/engine/xml/JRXmlLoader.load(JRXmlLoader.java:214)
      at net/sf/jasperreports/engine/JasperCompileManager.compileReport(JasperCompileManager.java:203)
      at com/dpwn/newops/reporting/server/service/ReceivingReportServiceImpl.generateWorkSheet(ReceivingReportServiceImpl.java:369)
      at com/dpwn/newops/reporting/server/service/ReceivingReportServiceImpl.generateAllWorkOrderReports(ReceivingReportServiceImpl.java:528)
      at com/dpwn/newops/reporting/server/facade/ReceivingReportFacadeImpl.generateAllWorkOrderReports(ReceivingReportFacadeImpl.java:87)
      at sun/reflect/GeneratedMethodAccessor8050.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
      at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
      at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:106)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      at org/springframework/aop/aspectj/MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
      at com/dpwn/newops/framework/server/remoting/TracingInterceptor.trace(TracingInterceptor.java:54)
      at sun/reflect/GeneratedMethodAccessor7989.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
      at org/springframework/aop/aspectj/AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
      at org/springframework/aop/aspectj/MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
      at com/dpwn/newops/framework/server/remoting/PerformanceInterceptor.logPerformance(PerformanceInterceptor.java:49)
      at sun/reflect/GeneratedMethodAccessor7988.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
      at org/springframework/aop/aspectj/AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
      at org/springframework/aop/aspectj/MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
      at com/dpwn/newops/framework/server/remoting/SecurityInterceptor.performAuthorization(SecurityInterceptor.java:71)
      at sun/reflect/GeneratedMethodAccessor7987.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
      at org/springframework/aop/aspectj/AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
      at org/springframework/aop/aspectj/AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
      at org/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy1051.generateAllWorkOrderReports(Ljava/util/List;Ljava/util/List;)Ljava/util/List;(Unknown Source)
      at sun/reflect/GeneratedMethodAccessor8050.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
          at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
      at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
      at org/springframework/remoting/support/RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy1052.generateAllWorkOrderReports(Ljava/util/List;Ljava/util/List;)Ljava/util/List;(Unknown Source)
      at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
      -- end of trace

      Memory usage report:
      Total mapped 4048716KB (reserved=5280KB)
      - Java heap 3145728KB (reserved=0KB)
      - GC tables 105228KB
      - Thread stacks 33732KB (#threads=169)
      - Compiled code 56768KB (used=53567KB)
      - Internal 1160KB
      - OS 11032KB
      - Other 279068KB
      - Java class data 414976KB (malloced=412194KB #470573 in 83896 classes)
      Not tracing sites.

      - Native memory tracking 1024KB (malloced=219KB #8)
      Not tracing sites.


      Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
      trace_alloc_sites=true to enable alloc site tracing.

      **************************************************************************
      * If you see this dump, please go to *
      * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
      * for troubleshooting information. *
      **************************************************************************

      ===== END DUMP ===============================================================