4 Replies Latest reply: Mar 30, 2011 11:50 PM by Akshay Koul RSS

    Weblogic Crashing

    824419
      Hi All,

      I am experincing problem with my weblogic which is currently running on linux. The major challenges that i am currently facing are

      -- weblogic hangs and freezes the application whenever many users login into the system and perform transaction.
      --- weblogic crashes frequently and also becomes unresponsive.

      I Have put the below parameters to tune the application but the application still become unresponsive.

      JAVA Memory arguments: -Xgc:gencon -XXcompactRatio:10 -XXcompactSetLimit:20000 -XXkeepAreaRatio:10 -XXcompressedRefs=0 -XXgcTrigger:50 -Xns:500m -Xms3072m -Xmx3072m -XXtlaSize:min=16k,preferred=512k -Xverbose:compaction -Xverbose:memory -Xverbose:gcpause -Xverbose:memdbg

      In addition below is the out the logfile before the weblogic crashes

      SUBSYSTEM = HTTP USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-101020 MACHINE = tquest TXID = CONTEXTID = TIMESTAMP = 1295432217706
      WatchAlarmType: AutomaticReset
      WatchAlarmResetPeriod: 30000
      >
      [INFO ][memdbg ] Adding 3 final handles from dying thread 12349 'process reaper'.
      [INFO ][memdbg ] Adding 3 final handles from dying thread 12353 'process reaper'.
      [INFO ][memdbg ] Adding 3 final handles from dying thread 12357 'process reaper'.
      [INFO ][memdbg ] Adding 3 final handles from dying thread 12361 'process reaper'.
      [INFO ][memdbg ] Adding 1 final handles from dying thread 12368 'oracle.dfw.dump.DumpManager - D'.
      [INFO ][memdbg ] Adding 1 final handles from dying thread 12370 'oracle.dfw.dump.DumpManager - D'.
      [INFO ][memdbg ] Adding 15 final handles from dying thread 12374 'oracle.dfw.dump.DumpManager - D'.
      <Jan 19, 2011 1:16:58 PM EAT> <Alert> <Diagnostics> <BEA-320016> <Creating diagnostic image in /u01/Oracle/Middleware/user_projects/domains/tquest/servers/AdminServer/adr/diag/ofm/tquest/AdminServer/incident/incdir_8423 with a lockout minute period of 1.>
      [INFO ][memdbg ] Thread 0xda3d1f98 ([ACTIVE] ExecuteThread: '16' fo) stopped roll forwarding (1 times, rfLimit: 100), will sleep and try again.
      [INFO ][memdbg ] Adding 3 final handles from dying thread 12372 'process reaper'.
      [INFO ][memdbg ] Adding 1 final handles from dying thread 12369 'oracle.dfw.dump.DumpManager - D'.
      [INFO ][memdbg ] Starting precleaning phase (OC3).
      [INFO ][memdbg ] Stopping of javathreads took 0.500 ms
      [INFO ][memdbg ] Starting final marking phase (OC4).
      [INFO ][nursery] KeepAreaStart: (nil) KeepAreaEnd: (nil)
      [INFO ][nursery] Young collection started. This YC is a part of OC#1334 final marking.
      [INFO ][memdbg ] Final marking phase promoted 38309 objects (2000K)
      [INFO ][memdbg ] Removing 20 permanent work packets from pool, now 1545 packets
      [INFO ][memdbg ] total concurrent mark time: 2393.200 ms
      [INFO ][memdbg ] ending marking phase
      [INFO ][memdbg ] starting concurrent sweeping phase
      [INFO ][nursery] Setting mmNurseryMarker[0] to 0x879e0638
      [INFO ][nursery] Setting mmNurseryMarker[1] to 0x8b28a1e0
      [INFO ][nursery] Nursery size increased from 0kb to 512000kb. Parts: 3161
      [INFO ][gcpause] old collection phase 4-5 pause time: 78.521000 ms, (start time: 7085.167 s)
      [INFO ][gcpause] (pause includes yc: 78.463 ms, compaction: 0.000 ms (no compaction), update ref: 0.000 ms)
      [INFO ][memdbg ] Restarting of javathreads took 0.138 ms
      [INFO ][memdbg ] Stopping of javathreads took 0.342 ms
      [INFO ][nursery] Nursery size remains at 512000kb. Parts: 3161
      [INFO ][gcpause] old collection phase 5 pause time: 0.847000 ms, (start time: 7085.257 s)
      [INFO ][memdbg ] Restarting of javathreads took 0.099 ms
      [INFO ][memdbg ] Stopping of javathreads took 0.399 ms
      [INFO ][memdbg ] total concurrent sweep time: 21.457 ms
      [INFO ][memdbg ] ending sweeping phase
      [INFO ][nursery] Nursery size remains at 512000kb. Parts: 3161
      [INFO ][memdbg ] Alloc Queue size after GC: 0, tlas: 0, oldest: 0
      [INFO ][compact] Compaction pause: 0.000 (target 247.808), update ref pause: 0.000 (target 247.808)
      [INFO ][compact] Updated 57719286 refs: 57719286 inside compaction area, and 0 outside (limit: 20000).
      [INFO ][compact] Compaction ended at index 104, object end address 0xa4800000.
      [INFO ][compact] Summary: 1334;32;104;12;0;0;0.000;247.808;0.000;247.808;57719286;20000
      [INFO ][memdbg ] gc-trigger is 50.000 % (locked)
      [INFO ][memdbg ] Nursery size after OC: 524288000
      [INFO ][gcpause] old collection phase 5-0 pause time: 0.745000 ms, (start time: 7085.265 s)
      [INFO ][memory ] 7082.850-7085.266: GC 2306058K->1460849K (3145728K), sum of pauses 121.436 ms
      [INFO ][memdbg ] Page faults before GC: 1, page faults after GC: 1, pages in heap: 786432
      [INFO ][finaliz] (OC) Pending finalizers 0->0
      [INFO ][memdbg ] Restarting of javathreads took 0.056 ms
      [INFO ][memdbg ] GC reason: GC trigger reached, cause: Heap too full
      [INFO ][memdbg ] Stopping of javathreads took 0.504 ms
      [INFO ][memdbg ] old collection 1335 started
      [INFO ][memdbg ] Alloc Queue size before GC: 0, tlas: 0, oldest: 0
      [INFO ][memdbg ] Compacting 12 heap parts at index 20 (type external) (exceptional false)
      [INFO ][compact] OC 1335: 12 parts (max 128), index 20. Type external, (exceptional false)
      [INFO ][compact] Area start: 0x26800000, end: 0x38800000
      [INFO ][memdbg ] Starting initial marking phase (OC1).
      [INFO ][nursery] KeepAreaStart: 0x8b28a1e0 KeepAreaEnd: 0x8f30fcc8
      [INFO ][nursery] Young collection started. This YC is a part of OC#1335 initial marking.
      [INFO ][nursery] Setting mmNurseryMarker[0] to 0x879e0638
      [INFO ][nursery] Setting mmNurseryMarker[1] to 0x8b28a1e0
      [INFO ][memdbg ] Initial marking phase promoted 1510 objects (91K)
      [INFO ][gcpause] old collection phase 1 pause time: 46.163000 ms, (start time: 7085.758 s)
      [INFO ][memdbg ] Restarting of javathreads took 0.135 ms
      [INFO ][memdbg ] Starting concurrent marking phase (OC2).
      [INFO ][memdbg ] Pointerset limit hit, compaction aborted.
      [INFO ][compact] Pointerset for thread '(GC Main Thread)' full.
      Tax Type FXD
      [INFO ][memdbg ] Adding 10 final handles from dying thread 12371 'oracle.dfw.dump.DumpManager - D'.
      [INFO ][memdbg ] Starting precleaning phase (OC3).
      [INFO ][memdbg ] Stopping of javathreads took 0.506 ms
      [INFO ][memdbg ] Starting final marking phase (OC4).
      [INFO ][nursery] KeepAreaStart: (nil) KeepAreaEnd: (nil)
      [INFO ][nursery] Young collection started. This YC is a part of OC#1335 final marking.
      [INFO ][memdbg ] Final marking phase promoted 73307 objects (5487K)
      [INFO ][memdbg ] Removing 20 permanent work packets from pool, now 1525 packets
      [INFO ][memdbg ] total concurrent mark time: 1711.340 ms
      [INFO ][memdbg ] ending marking phase
      [INFO ][memdbg ] starting concurrent sweeping phase
      [INFO ][nursery] Setting mmNurseryMarker[0] to 0x86242fe8
      [INFO ][nursery] Setting mmNurseryMarker[1] to 0x89c0c110
      [INFO ][nursery] Nursery size increased from 0kb to 512000kb. Parts: 3082
      [INFO ][gcpause] old collection phase 4-5 pause time: 71.587000 ms, (start time: 7087.399 s)
      [INFO ][gcpause] (pause includes yc: 71.537 ms, compaction: 0.000 ms (no compaction), update ref: 0.000 ms)
      [INFO ][memdbg ] Restarting of javathreads took 0.166 ms
      [INFO ][memdbg ] Stopping of javathreads took 0.452 ms
      [INFO ][nursery] Nursery size remains at 512000kb. Parts: 3082
      [INFO ][gcpause] old collection phase 5 pause time: 0.845000 ms, (start time: 7087.477 s)
      [INFO ][memdbg ] Restarting of javathreads took 0.076 ms
      [INFO ][memdbg ] Stopping of javathreads took 0.443 ms
      [INFO ][memdbg ] total concurrent sweep time: 13.587 ms
      [INFO ][memdbg ] ending sweeping phase
      [INFO ][nursery] Nursery size remains at 512000kb. Parts: 3082
      [INFO ][memdbg ] Alloc Queue size after GC: 0, tlas: 0, oldest: 0
      [INFO ][compact] Compaction pause: 0.000 (target 247.808), update ref pause: 0.000 (target 247.808)
      [INFO ][compact] Updated 57719409 refs: 57719286 inside compaction area, and 123 outside (limit: 20000).
      [INFO ][compact] Compaction ended at index 104, object end address 0xa4800000.
      [INFO ][compact] Summary: 1335;20;104;12;0;0;0.000;247.808;0.000;247.808;57719409;20000
      [INFO ][memdbg ] gc-trigger is 50.000 % (locked)
      [INFO ][memdbg ] Nursery size after OC: 524288000
      [INFO ][gcpause] old collection phase 5-0 pause time: 0.675000 ms, (start time: 7087.483 s)
      [INFO ][memory ] 7085.758-7087.484: GC 2139390K->1424733K (3145728K), sum of pauses 119.270 ms
      [INFO ][memdbg ] Page faults before GC: 1, page faults after GC: 1, pages in heap: 786432
      [INFO ][finaliz] (OC) Pending finalizers 0->0
      [INFO ][memdbg ] Restarting of javathreads took 0.032 ms
      SELECT DISTINCT GIN_SCREEN_DST_LEVEL.* FROM GIN_SCREEN_DST_LEVEL WHERE DSL_SCREEN_ID = 4 AND DSL_LEVEL_NO = 1
      <Jan 19, 2011 1:17:03 PM EAT> <Warning> <oracle.adfinternal.view.faces.renderkit.rich.FormRenderer> <ADF_FACES-30124> <Multiple forms detected on viewId: /alterPolicyRisk.jspx. Rich client currently has some limitations in dealing with multiple forms.>
      [INFO ][memory ] Allocation of 1352 bytes failed for heap "<unnamed>"
      [INFO ][memory ] TLA bailout requested (heap=0xed70e1f0)!
      [INFO ][memory ] TLA unwind thread links.
      [INFO ][memory ] Throwing OutOfMemory: CG(o0) [oracle/binding/metaimpl/BaseAttributeDefinition.<init>(Ljava/lang/String;Loracle/binding/meta/NamedDefinition;Ljava/lan
      [INFO ][memory ] Allocation of 264 bytes failed for heap "<unnamed>"
      [INFO ][memory ] TLA bailout requested (heap=0xec76d1f0)!
      [INFO ][memory ] TLA unwind thread links.
      [ERROR] JRockit Fatal Error: Unspecified error (52)
      [ERROR] As of this writing noone should EVER use native stack unwind except myself.
      [JRockit] ERROR: The JVM has crashed. Writing crash information to /u01/Oracle/Middleware/user_projects/domains/tquest/jrockit.8655.dump.


      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 0 days, 01:58:11 on Wed Jan 19 13:17:05 2011

      **********************************************************
      * If you see this dump, please go to *
      * http://edocs.bea.com/jrockit/go2troubleshooting.html *
      * for troubleshooting information. *
      **********************************************************

      Additional information is available in:
      /u01/Oracle/Middleware/user_projects/domains/tquest/jrockit.8655.dump
      No snapshot file (core dump) will be created because core dumps have been
      disabled. To enable core dumping, try "ulimit -c unlimited"
      before starting JRockit again.

      Error Message: Unspecified error [52]
      Signal info : si_signo=11, si_code=2 si_addr=(nil)
      Fatal Error : As of this writing noone should EVER use native stack unwind except myself.
      Version : BEA JRockit(R) R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-ia32
      GC Strategy : Mode: static, with strategy: genconcon
      GC Status : OC is not running. Last finished OC was OC#1335.
      : YC is not running. Last finished YC was YC#99.
      YC History : Ran 0 YCs before OC#1331.
      : Ran 0 YCs before OC#1332.
      : Ran 0 YCs before OC#1333.
      : Ran 0 YCs before OC#1334.
      : Ran 0 YCs before OC#1335.
      : Ran 0 YCs since last OC.
      YC Promotion : Last YC successfully promoted all objects
      Heap : 0x8800000 - 0xc8800000 (Size: 3072 MB)
      Compaction : 0x26800000 - 0x38800000 (Current compaction type: nocompaction)
      NurseryList : 0x68800000 - 0x8d681d78
      KeepArea : (no keeparea in use)
      NurseryMarker: [ 0x86242fe8,  0x89c0c110 ]
      CompRefs : References are 32-bit.
      CPU : Intel (null) (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 EM64T
      Number CPUs : 8
      Tot Phys Mem : 16827752448 (16048 MB)
      OS version : Red Hat Enterprise Linux Server release 4.4 (Tikanga)
      Linux version 2.6.18-92.el5 (brewbuilder@ls20-bc2-13.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:15 EDT 2008 (x86_64)
      Thread System: NPTL
      Java locking : Lazy unlocking enabled (class banning) (transfer banning)
      State : JVM is running
      Command Line : -Xgc:gencon -XXcompactRatio:10 -XXcompactSetLimit:20000 -XXkeepAreaRatio:10 -XXcompressedRefs=0 -XXgcTrigger:50 -Xns:500m -Xms3072m -Xmx3072m -XXtlaSize:min=16k,preferred=512k -Xverbose:compaction -Xverbose:memory -Xverbose:gcpause -Xverbose:memdbg -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/u01/Oracle/Middleware/wlserver_10.3 -Dwls.home=/u01/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/u01/Oracle/Middleware/wlserver_10.3/server -Ddomain.home=/u01/Oracle/Middleware/user_projects/domains/tquest -Dcommon.components.home=/u01/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/Oracle/Middleware/user_projects/domains/tquest/config/fmwconfig -Doracle.server.config.dir=/u01/Oracle/Middleware/user_projects/domains/tquest/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/Oracle/Middleware/user_projects/domains/tquest/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/u01/Oracle/Middleware/user_projects/domains/tquest/config/fmwconfig/carml -Digf.arisidstack.home=/u01/Oracle/Middleware/user_projects/domains/tquest/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/product/10.2.0/db_1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/product/10.2.0/db_1/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/u01/Oracle/Middleware/patch_wls1032/profiles/default/sysext_manifest_classpath:/u01/Oracle/Middleware/patch_jdev1111/profiles/default/sysext_manifest_classpath -Dsun.java.launcher=SUN_STANDARD weblogic.Server
      java.home : /usr/jrmc-3.0.3-1.6/jre
      j.class.path : /jarlib/antlr.jar:/jarlib/jasperreports-3.5.1.jar:/jarlib/iText-2.1.3_001.jar:/jarlib/poi-3.2-FINAL.jar:/jarlib/jakarta-poi-1.5.1-final-2002615.jar::/u01/Oracle/Middleware/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/Oracle/Middleware/patch_jdev1111/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/jrmc/lib/tools.jar:/u01/Oracle/Middleware/utils/config/10.3/config-launch.jar:/u01/Oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/Oracle/Middleware/modules/features/weblogic.server.modules_10.3.2.0.jar:/u01/Oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/u01/Oracle/Middleware/modules/org.apache.ant_1.7.0/lib/ant-all.jar:/u01/Oracle/Middleware/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar:/u01/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/Oracle/Middleware/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/u01/Oracle/Middleware/wlserver_10.3/server/lib/xqrl.jar
      j.lib.path : /usr/jrmc-3.0.3-1.6/jre/lib/i386/jrockit:/usr/jrmc-3.0.3-1.6/jre/lib/i386:/usr/jrmc-3.0.3-1.6/jre/../lib/i386:/u01/Oracle/Middleware/patch_wls1032/profiles/default/native:/u01/Oracle/Middleware/patch_jdev1111/profiles/default/native:/u01/Oracle/Middleware/wlserver_10.3/server/native/linux/i686:/u01/Oracle/Middleware/wlserver_10.3/server/native/linux/i686/oci920_8
      JAVA_HOME : /usr/jrmc
      JAVAOPTIONS: <not set>
      LD_LIBRARY_PATH: /usr/jrmc-3.0.3-1.6/jre/lib/i386/jrockit:/usr/jrmc-3.0.3-1.6/jre/lib/i386:/usr/jrmc-3.0.3-1.6/jre/../lib/i386:/u01/Oracle/Middleware/patch_wls1032/profiles/default/native:/u01/Oracle/Middleware/patch_jdev1111/profiles/default/native:/u01/Oracle/Middleware/wlserver_10.3/server/native/linux/i686:/u01/Oracle/Middleware/wlserver_10.3/server/native/linux/i686/oci920_8
      LD_ASSUME_KERNEL: <not set>
      C Heap : Good; no memory allocations have failed
      Method : oracle/adf/model/binding/DCUtil.findSpelObject(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory : 2 OutOfMemoryErrors have occured

      Registers (from ThreadContext: 0xf5ab4b90 / OS context: 0xf5ab4c8c):
      eax = 00001267 ecx = 00000000 edx = 0000004b ebx = f7f483cc
      esp = f5ab4f80 ebp = f5ab4f98 esi = 00000034 edi = f5ab4fe8
      es = 0000002b cs = 656e0023 ss = 5245002b ds = 0000002b
      fs = 00000003 gs = 00000063
      eip = f7d8f3a5 eflags = 00000282

      Stack:
      (* marks the word pointed to by the stack pointer)
      f5ab4f80: f7f56020* 00000200 f7f483cc f5ab4fe8 f5ab4fe8 f5ab4fa4
      f5ab4f98: f5ab4fb8 f7f092b4 00000034 f7f483cc f5ab4fe8 f5ab52a0
      f5ab4fb0: f57f8490 fffffef8 f5ab4fd8 f7f092df 00000034 f7f483cc
      f5ab4fc8: f5ab4fe8 fffffef8 00010000 ec76d008 f5ab4fe8 f7ef989b

      Code:
      (* marks the word pointed to by the instruction pointer)
      f7d8f374: 4c892074 458b0824 2404c710 f7f56020 0c244489 000200b8
      f7d8f38c: 24448900 969ae804 01b80017 a3000000 f7f56004 001267b8
      f7d8f3a4: 0000a300* 04c70000 00003f24 9e8ae800 768d0017 27bc8d00
      f7d8f3bc: 00000000 e589fc55 53c03157 b9e87d8d 00000004 00c0ec81

      Loaded modules:
      (* denotes the module causing the exception)
      08048000-08058253 /usr/jrmc-3.0.3-1.6/bin/java
      ffffe000-ffffe7d7 /usr/jrmc-3.0.3-1.6/bin/java
      005ee000-0060037f /lib/libpthread.so.0
      005c5000-005e9167 /lib/libm.so.6
      00607000-00608d0b /lib/libdl.so.2
      00480000-005bc21b /lib/libc.so.6
      00462000-0047b0f3 /lib/ld-linux.so.2
      f7cf9000-f7f50367 */usr/jrmc-3.0.3-1.6/jre/lib/i386/jrockit/libjvm.so
      009c8000-009ce47b /lib/librt.so.1
      f7fe3000-f7feb7cb /lib/libnss_files.so.2
      f7fcf000-f7fd99bb /usr/jrmc-3.0.3-1.6/jre/lib/i386/libverify.so
      f71dd000-f71ffdf7 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libjava.so
      00791000-007a3c1f /lib/libnsl.so.1
      f6136000-f613bef0 /usr/jrmc-3.0.3-1.6/jre/lib/i386/native_threads/libhpi.so
      f5b4c000-f5b5a3e4 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libzip.so
      f5870000-f5882b38 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libnet.so
      f5705000-f570b424 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libnio.so
      f3097000-f309c666 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libmanagement.so
      f3028000-f3031357 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libjmapi.so
      f7fcc000-f7fcdde4 /u01/Oracle/Middleware/wlserver_10.3/server/native/linux/i686/libwlfileio2.so
      f6143000-f61469b7 /lib/libnss_dns.so.2
      0077c000-0078a76f /lib/libresolv.so.2
      ea5da000-ea654966 /usr/jrmc-3.0.3-1.6/jre/lib/i386/libawt.so
      f61d8000-f61dbfa8 /usr/jrmc-3.0.3-1.6/jre/lib/i386/headless/libmawt.so


      "(Code Optimization Thread 1)" id=5 idx=0x34 tid=8668 lastJavaFrame=0xfffffffc

      Stack 0: start=0xf5a94000, end=0xf5ab8000, guards=0xf5a99000 (ok), forbidden=0xf5a97000
      Thread Stack Trace:
      at dumpForceDump+117()@0xf7d8f3a5
      at vmFatalErrorMsgV+84()@0xf7f092b4
      at vmFatalErrorMsg+31()@0xf7f092df
      at tla_linknodefunc+27()@0xf7ef989b
      at vmtiUnlinkData+55()@0xf7f0b207
      at vmtLinkUnwind+50()@0xf7f0b262
      at tlaBail+132()@0xf7ef9704
      at tla_fail_alloc+68()@0xf7ef9874
      at handle_alloc_failure+41()@0xf7e7c059
      at mspace_malloc+1286()@0xf7e7d076
      at tlaMallocOrBail+30()@0xf7ef978e
      at buf_assure_space+157()@0xf7dd530d
      at irOpCopy+612()@0xf7dc9094
      at irAppendIR+659()@0xf7d4fc23
      at insert_called_code+444()@0xf7dbf3cc
      at inline_call+314()@0xf7dc05ba
      at cgInlineSmallAndSpecial+460()@0xf7dc12cc
      at cgInlineAll+131()@0xf7dc1473
      at optmanOptimizeMIR+217()@0xf7e8de79
      at generateMethodWithStage+94()@0xf7d6fa1e
      at cmgrGenerateMethodFromPhase+215()@0xf7d708f7
      at cmgrGenerateNormalMethod+87()@0xf7d70967
      at cmgrGenerateCode+383()@0xf7d70b1f
      at generate_code2+937()@0xf7e587b9
      at codegenThread+745()@0xf7e59749
      at tsiCallStartFunction+81()@0xf7e30121
      at tsiThreadStub+140()@0xf7e3128c
      at ptiThreadStub+18()@0xf7e9f5d2
      at start_thread+203()@0x5f345b
      at __clone+94()@0x550c4e
      -- Java stack --
      Pending Exception:
      java/lang/OutOfMemoryError: CG(o0) [oracle/binding/metaimpl/BaseAttributeDefinition.<init>(Ljava/lang/String;Loracle/binding/meta/NamedDefinition;Ljava/lan
      --- End of stack trace

      Extended, platform specific info:
      libc release: 2.5-stable
      Elf headers:
      libc ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00495f30 PHOFF: 00000034 SHOFF: 00186698 EF: 0x0 HS: 52 PS: 32 PHN; 10 SS: 40 SHN: 75 STIDX: 74
      libpthread ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 005f27f0 PHOFF: 00000034 SHOFF: 0001e4f0 EF: 0x0 HS: 52 PS: 32 PHN; 9 SS: 40 SHN: 40 STIDX: 39
      libjvm ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 0004bf70 PHOFF: 00000034 SHOFF: 012c54a4 EF: 0x0 HS: 52 PS: 32 PHN; 4 SS: 40 SHN: 29 STIDX: 26

      **********************************************************
      * If you see this dump, please go to *
      * http://edocs.bea.com/jrockit/go2troubleshooting.html *
      * for troubleshooting information. *
      **********************************************************

      ===== END DUMP ===============================================================


      /u01/Oracle/Middleware/user_projects/domains/tquest/bin/startWebLogic.sh: line 194: 8655 Aborted


      kindly Assist .I would really appreciate all advice in resolving the issue once and for all.
      .
        • 1. Re: Weblogic Crashing
          René van Wijk
          I think it is a good idea to clean the JVM arguments to something like:

          when you want to optimize for response time: -Xms512m -Xmx512m -Xns256m -XXkeepAreaRatio:25 -Xgcprio:pausetime -XpauseTarget:200ms
          or when you want to optimize for throughput: -Xms512m -Xmx512m -Xss128k -Xgcprio:throughput

          The options Xms, Xms and Xns can be configured to your preference. Note however that when you are running on 32bit machine
          that the maximum (Xmx) cannot be higher than 1750M. You must also account for the memory usage of the JVM itself (1250M to 1500M
          is probably a good value on a 32bit system).

          To check which linux version you are using you can use uname -a.
          • 2. Re: Weblogic Crashing
            824419
            Thanks will try this out will give u a feedback on the progress.
            • 3. Weblogic Crashing (Jayashankar- OpSource)
              844812
              I am running weblogic 9.2.0.0 on Windows(R) Server 2003, Standard Editionerver 2003, Standard Edition. we have been observing the server crashing at time and also in hung state; I've found some crash dumps jrockit.dump & jrockit.mdmp. Please nore we are using java version 1.5

              Here is the complete Java version details:
              ==============================
              java version "1.5.0_04"
              Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
              BEA JRockit(R) (build R26.0.0-189_CR269406-59389-1.5.0_04-20060322-1120-win-ia32
              , )

              Here is the OS info
              =================
              OS Name: Microsoft(R) Windows(R) Server 2003, Standard Edition
              OS Version: 5.2.3790 Service Pack 2 Build 3790

              I am pasting the content from jrockit.dump, jrockit.mdmp being very huge abut 600 MB please let me know were I can upload the same. Also based on my previous dump analysis we have implemented a JVM flag "-Xverify:all" to take care of this issue based on the recommendation that I found here in this link http://www.coderanch.com/t/425380/BEA-Weblogic/Weblogic-crash-after-Illegal-memory


              ===== BEGIN DUMP =============================================================
              JRockit dump produced after 5 days, 21:49:19 on Mon Feb 28 11:07:14 2011

              Additional information is available in:
              E:\bea92\USER_P~1\domains\DBO2DO~1\jrockit.5544.dump
              E:\bea92\USER_P~1\domains\DBO2DO~1\jrockit.5544.mdmp
              If you see this dump, please open a support case with BEA and
              supply as much information as you can on your system setup and
              the program you were running. You can also search for solutions
              to your problem at http://forums.bea.com in
              the forum jrockit.developer.interest.general.

              Error Message: Illegal memory access. [54]
              Exception Rec: EXCEPTION_ACCESS_VIOLATION (c0000005) at 0x008485C0 - memory at 0x00000004 could not be read.
              Minidump : Wrote mdmp. Size is 461MB
              Version : BEA JRockit(R) R26.0.0-189_CR269406-59389-1.5.0_04-20060322-1120-win-ia32
              GC : System optimized over throughput (initial strategy singleparpar)
              : GC strategy for GC 11045 was singleparpar
              : GC strategy for GC 11046 was singleparpar
              : GC strategy for GC 11047 was singleparpar
              : GC strategy for GC 11048 was singleparpar
              : GC strategy for GC 11049 was singleparpar
              : mmHeap->data = 0x00C00000, mmHeap->top = 0x40C00000
              : mmStartCompaction = 0x34C00000, mmEndCompaction = 0x38C00000
              : Heap is not included in dumps.
              CPU : AMD Pentium 4 SSE SSE2 EM64T
              Number CPUs : 4
              Tot Phys Mem : 4118257664 (3927 MB)
              OS version : Microsoft Windows Server 2003 version 5.2 Service Pack 2 (Build 3790)
              Thread System: Windows Threads
              State : JVM is running
              Command Line : -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=7777,server=y,suspend=n -Djava.compiler=NONE -Xms512m -Xmx1024m -ea -da:com.bea... -da:javelin... -da:weblogic... -ea:com.bea.wli... -ea:com.bea.broker... -ea:com.bea.sbconsole... -Dplatform.home=E:\bea92\WEBLOG~1 -Dwls.home=E:\bea92\WEBLOG~1\server -Dwli.home=E:\bea92\WEBLOG~1\integration -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=E:\bea92\patch_weblogic920\profiles\default\sysext_manifest_classpath -Dcom.sun.management.jmxremote.port=12347 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dweblogic.Name=dbo2Server -Dweblogic.management.username=weblogic -Djava.security.policy=E:\bea92\WEBLOG~1\server\lib\weblogic.policy -Dweblogic.management.password=weblogic_l1v3 -Dweblogic.management.startmode=WinSvc
              JAVA_HOME : E:\bea92\jrockit90_150_04
              java.home : E:\bea92\JROCKI~1\jre
              j.class.path : E:\bea92\patch_weblogic920\profiles\default\sys_manifest_classpath\weblogic_patch.jar;E:\bea92\JROCKI~1\lib\tools.jar;E:\bea92\WEBLOG~1\server\lib\weblogic_sp.jar;E:\bea92\WEBLOG~1\server\lib\weblogic.jar;E:\bea92\WEBLOG~1\server\lib\webservices.jar;;E:\bea92\patch_weblogic920\profiles\default\sys_manifest_classpath\weblogic_patch.jar;E:\bea92\JROCKI~1\lib\tools.jar;E:\bea92\WEBLOG~1\server\lib\weblogic_sp.jar;E:\bea92\WEBLOG~1\server\lib\weblogic.jar;E:\bea92\WEBLOG~1\server\lib\webservices.jar;E:\bea92\WEBLOG~1\server\lib\jtds-1.2.2.jar;E:\bea92\WEBLOG~1\server\lib\xqrl.jar;;E:\bea92\WEBLOG~1\integration\lib\util.jar;
              j.lib.path : E:\bea92\WEBLOG~1\server\bin;.;C:\WINDOWS\system32;C:\WINDOWS;E:\bea92\WEBLOG~1\server\native\win\32;E:\bea92\WEBLOG~1\server\bin;E:\bea92\JROCKI~1\jre\bin;E:\bea92\JROCKI~1\bin;E:\bea92\WEBLOG~1\server\native\win\32\oci920_8;;E:\bea92\jrockit90_150_04\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;E:\apache-ant-1.7.1\bin;
              C Heap : 1 memory allocations have failed
              : First failure was a mmMalloc of 16777216 bytes
              : Last failure was a mmMalloc of 16777216 bytes
              Method : jsp_servlet/_pages/_general/__resetcaches.<clinit>()V
              StackOverFlow: 0 StackOverFlowErrors have occured

              Registers (from ThreadContext: 0x5E29F064 / OS context: 0x5E29F430):
              eax = 00000000 ecx = 00000000 edx = 5e29fe70 ebx = 5e2dcfd8
              esp = 5e29f6fc ebp = 00000000 esi = 00000000 edi = 66b4cc44
              es = 00000023 cs = 0000001b ss = 00000023 ds = 00000023
              fs = 0000003b gs = 00000000
              eip = 008485c0 eflags = 00010202

              Stack:
              (* marks the word pointed to by the stack pointer)
              5e29f6fc: 00848ada* 41f95f00 009842d5 00000000 5e29f72c 66b4caf8
              5e29f714: 66b4cc44 00000000 32309a00 00000000 00000000 02628108
              5e29f72c: 00000000 5e29fe70 00000000 66b4caf8 5e2dd03c 00984400
              5e29f744: 66b4cc44 00000000 00000000 5e29f758 7e350e05 66b4cc44
              5e29f75c: 5e29f774 32309e48 4152ca20 7e350e00 6fabf4ec 5e29f758
              5e29f774: 32309a00 66b4caf8 00000000 32309a00 7e330495 32309e30
              5e29f78c: 7e350d50 32309b98 32309e78 32309b98 7e350487 32309b98
              5e29f7a4: 32309e30 00000000 7e350436 32309e30 32309b98 00000000
              5e29f7bc: 32309e30 769e3cd8 00000009 32309a00 02628108 32309b98
              5e29f7d4: 7ecae525 32309e30 32309a00 096c6e80 32309e30 32309a88
              5e29f7ec: 00000001 32309a00 7ecac2aa 00000000 323a9310 322fa248
              5e29f804: 00000001 32309a00 00000000 02578f60 7eca92eb 323a7578
              5e29f81c: 00000001 322fa390 0ee4af68 322fa248 3233a2c8 323a9310
              5e29f834: 7ecac10a 323a9310 7e32218f 323a9311 322fa248 322fa248
              5e29f84c: 7ec9fd75 3233a7f8 3233a7f8 00000000 7e85a8b0 323a9a08
              5e29f864: 323a9310 3233a2c8 3233a7f8 00000000 01037b30 01037b30
              5e29f87c: 323a9aa0 01037b30 7e873240 3233a2c8 3233a2c8 3233a7f8
              5e29f894: 323a9311 3233a7f8 323a7578 323a9310 322fa248 323a9310
              5e29f8ac: 3233a2c8 323a9310 7ecac06f 3233a7f8 00000000 096d12d8
              5e29f8c4: 323a9380 3233a2c8 3233a2c8 3233a7f8 00000000 3233a7f8
              5e29f8dc: 096c6e81 323a9310 323a9380 099ea7b0 096c6e80 099ea7b0
              5e29f8f4: 09a6bd48 00000000 7eca0096 3233a2c8 3233a2c8 3233a7f8
              5e29f90c: 00000000 09a6bd48 096d12d8 3233a2c8 3233a7f8 00000000
              5e29f924: 09a6bd48 6d059b40 0000012e 07ded160 096d12d8 3233a2c8
              5e29f93c: 3233a2c8 3233a7f8 00000001 3233a7f8 00000000 02546678
              5e29f954: 323a74d8 323a73d0 3233a7f8 3233a2c8 7ec9f40b 3233a7f8
              5e29f96c: 00000000 096d12d8 3233a2c8 3233a7f8 7ecc05ca 3233a7f8
              5e29f984: 3233a7f8 323a71b8 00000000 3233a2c8 3233a7f8 3233a7f8
              5e29f99c: 00000001 00000000 00e45d58 3233a7f8 7ecbedd2 3233a7f8
              5e29f9b4: 3233a7f8 323a71b8 3233a2c8 3233a7f8 3233a2c8 3233a7f8
              5e29f9cc: 00000000 00000000 00000000 00000000 00000000 00000000
              5e29f9e4: 00000000 323a7338 3233c318 00000001 00000000 00000000
              5e29f9fc: 00000000 096c6e80 00000000 00000000 00000000 00000000
              5e29fa14: 3233a7f8 00000000 00000001 096c6e80 096c6e80 00000001
              5e29fa2c: ffffffff 00e45d58 323a7118 3233a7f8 0ee4af68 096c6e80
              5e29fa44: 323a7118 3233a7f8 7ecbe23c 3233a7f8 06c8b200 323a7118
              5e29fa5c: 3233a2c8 3233a7f8 323a71b8 05cb3a68 3233a7f8 06c8b200
              5e29fa74: 7ecad191 3233a2c8 3233a7f8 06c8b200 3233a2c8 3233a7f8
              5e29fa8c: 05cb39e8 05cb3a68 323a7118 09a527f0 323a6498 06c8b1d8
              5e29faa4: 06c8b200 3233a2c8 7eca2a8f 3233a7f8 05cb39e8 3233efd8
              5e29fabc: 05ca4b58 06c8b200 3233a2c8 3233a7f8 3233e300 05ca4b58
              5e29fad4: 3233efd8 323a58d8 05cb39e8 09a527f0 3233e318 05fda308
              5e29faec: 3233a2c8 3233a7f8 7eca18a5 3233a7f8 05fda308 3233a2c8
              5e29fb04: 3233a7f8 66b4caf8 3233a7f8 ffffffff 7eca1629 3233a7f8
              5e29fb1c: 05fda308 3233a2c8 3233a7f8 6fabf4ec 7e32de7d 00000000
              5e29fb34: 7698191c 7698191c 7e334285 00ff5308 3233a7f8 ffffffff
              5e29fb4c: 7eca15e6 3233a7f8 05fda308 3233a2c8 3233a7f8 05fda308
              5e29fb64: 7eca1453 3233a7f8 00000000 05fda308 3233a2c8 3233a7f8
              5e29fb7c: 00ff5308 ffffffff ffffffff 026c9150 7e824dac 07df33f0
              5e29fb94: 7e33014d 07df33f0 07e05620 096d1288 7eca109c 3233a7f8
              5e29fbac: 05fda308 3233a2c8 3233a7f8 3233a2c8 3233a7f8 02193ba0
              5e29fbc4: 7eca0fcf 096d1288 7eca0f93 3233a7f8 3233d210 ffffffff
              5e29fbdc: 07e05620 3233d210 7eca0d0f 3233a2c8 3233a7f8 05fda308
              5e29fbf4: 096d1288 07e05620 3233a2c8 3233a2c8 3233a2c8 3233a7f8
              5e29fc0c: 3233a7f8 05fda308 3233a2c8 02193ba0 7eca06bd 02193ba0
              5e29fc24: 05fda308 3233a2c8 7ec9fef6 3233a2c8 3233a2c8 3233a7f8
              5e29fc3c: 3233a7f8 05fda308 096d1288 3233a2c8 3233a7f8 00000000
              5e29fc54: 05fda308 6d059ac5 0000012e 07ded160 096d1288 3233a2c8
              5e29fc6c: 3233a2c8 3233a7f8 00000000 3233a7f8 00000000 096cd8b8
              5e29fc84: 3233d1f8 3233c870 07ded160 3233a7f8 7ec9f40b 3233a7f8
              5e29fc9c: 00000000 096d1288 3233a2c8 3233a7f8 7ec9eef3 3233a7f8
              5e29fcb4: 3233a7f8 3233d108 7e3ff9d6 096d1288 3233a2c8 01037b30
              5e29fccc: 01037b30 01036dc8 7e6250e9 01036dc8 01036dc8 01037b30
              5e29fce4: 3233d108 00000000 116ddda8 3233a2c8 3233a2c8 3233d108
              5e29fcfc: 3233d108 01036dc8 7e625040 3233d108 01037b30 01036dc8
              5e29fd14: 3233d108 01036dc8 7ec9b1c5 3233d108 3233a7f8 096d1288
              5e29fd2c: 096c6e80 3233a2c8 3233a7f8 00000001 00000011 3233a2c8
              5e29fd44: 3233d108 00000000 66b4caf8 3233a2c8 7ec9a51f 3233a7f8
              5e29fd5c: 3233a2c8 096c6e80 3233a2c8 3233a7f8 0ee4af68 00c67800
              5e29fd74: 00000000 07ded0b8 0ee4af68 3233a5e8 66b4caf8 32339168
              5e29fd8c: 7ec99478 3233a7f8 0000012e 3233a2c8 096c6e80 0ee4af68
              5e29fda4: 7e5543f6 0ee4af68 66b4caf8 32339168 7e5542e6 6d059ac5
              5e29fdbc: 0000012e 0ee4af68 3233a2c8 3233a2c8 66b4caf8 32339168
              5e29fdd4: 7e54ad70 32339168 0ee4af68 00000000 394c98b6 fffffffc
              5e29fdec: 7e321b01 fffffffc 5e29fe54 5e29fe9c 7e54acf0 00980c27
              5e29fe04: 66b4cc44 66b4caf8 5e29fe34 5e29fe9c 00978604 66b4caf8
              5e29fe1c: 7e54acf0 5e29fe9c 5e29ff34 00000004 00000004 5e29fe9c
              5e29fe34: 40dcfe98 009455ec 7e321ac0 66b4caf8 7e54acf0 5e29fe9c
              5e29fe4c: 5e29ff34 66b4cc44 66b4caf8 76ba38d8 5e29ff34 00000000
              5e29fe64: 00000000 00000001 40d88974 00000000 5e29ff0c 00000001
              5e29fe7c: 40d88974 00000006 00000001 00000001 00000001 00000000
              5e29fe94: 00000001 00000000 66b4caf8 00000000 00000000 00000000
              5e29feac: 00000000 00000000 00000000 00000000 00906dc7 7795ae88
              5e29fec4: 7795c080 008ac06c 66b4cc44 7795ae88 7795c080 5e29ff0c
              5e29fedc: 008ac1b2 66b4cc44 76ba38d8 435b0ca8 66b4caf8 00000000
              5e29fef4: 5e29ff4c 009452f0 5e29ff34 66b4cc44 5e29ff64 66b4caf8

              Code:
              (* marks the word pointed to by the instruction pointer)
              008484c0: 04418b0c e8505756 00008b34 e857388b ffff8ebc c483f08b
              008484d8: 75f68508 b85e5f0a 00000004 53c3595d 1024548d 24448d52
              008484f0: 57555020 c206e856 d88b000b 8514c483 8b3775db 74f68536
              00848508: 084e8b26 468b118b 52088b04 028b168b 51088b51 9a0a1468
              00848520: 00685300 e8000400 00010304 5b18c483 04b85e5f 5d000000
              00848538: 448bc359 c0851c24 558b1174 57535200 1c02e856 c483000d
              00848550: 8b1beb10 85102444 501374c0 1c24448b 6a04488b 15e85101
              00848568: 8300008a 538b0cc4 24448b04 04488b18 51e85152 8300008a
              00848580: 5f5b08c4 0001b85e 595d0000 ccccccc3 8d18518b 500b1841
              00848598: 6a157504 418b5001 90685104 50008484 fecaf3e8 14c483ff
              008485b0: ccccccc3 cccccccc cccccccc cccccccc 85044e8b* 560c75c9
              008485c8: ffcf53e8 04c483ff f6c3c033 75012846 18568b21 0b18468d
              008485e0: 12750450 5650016a 84849068 ade85100 83fffeca 4e8314c4
              008485f8: 3d830128 00a32654 8b377c03 74c08506 084e8b31 5104568b
              00848610: 5208488b 488b118b 52008b04 088b118b 99e224a1 118b5200
              00848628: 40c08352 9a0a4c68 79e85000 830000fd 01b81cc4 c3000000
              00848640: 5614ec83 448df08b 68500424 009e1fc8 0d77bbe8 180d8b00
              00848658: 89009e20 0d8b104e 009e201c 20183589 8941009e 9e201c0d
              00848670: 04468b00 8b08568b 9e202035 8dd02b00 500c2444 c868f203
              00848688: 89009e1f 9e202035 6d7ae800 c483000d c4835e10 ccccc314
              008486a0: 7c8b5756 478d1024 31e85008 8b000a83 04c483f0 f685c033
              008486b8: 4c8b4474 0e890c24 d18bcf8b 02e9c153 8b085e8d 8babf3fb

              Loaded modules:
              (* denotes the module causing the exception)
              00400000-0040dfff E:\bea92\WEBLOG~1\server\bin\beasvc.exe
              7c800000-7c8c1fff C:\WINDOWS\system32\ntdll.dll
              77e40000-77f41fff C:\WINDOWS\system32\kernel32.dll
              7d1e0000-7d27bfff C:\WINDOWS\system32\ADVAPI32.dll
              77c50000-77ceffff C:\WINDOWS\system32\RPCRT4.dll
              76f50000-76f62fff C:\WINDOWS\system32\Secur32.dll
              71bd0000-71be0fff C:\WINDOWS\system32\MPR.dll
              77380000-77410fff C:\WINDOWS\system32\USER32.dll
              77c00000-77c48fff C:\WINDOWS\system32\GDI32.dll
              77ba0000-77bf9fff C:\WINDOWS\system32\MSVCRT.dll
              76290000-762acfff C:\WINDOWS\system32\IMM32.DLL
              00820000-00a44fff *E:\bea92\JROCKI~1\jre\bin\jrockit\jvm.dll
              76aa0000-76accfff C:\WINDOWS\system32\WINMM.dll
              71c00000-71c16fff C:\WINDOWS\system32\WS2_32.dll
              71bf0000-71bf7fff C:\WINDOWS\system32\WS2HELP.dll
              7c340000-7c395fff E:\bea92\JROCKI~1\jre\bin\MSVCR71.dll
              71bc0000-71bc7fff C:\WINDOWS\system32\rdpsnd.dll
              771f0000-77200fff C:\WINDOWS\system32\WINSTA.dll
              71c40000-71c96fff C:\WINDOWS\system32\NETAPI32.dll
              76b70000-76b7afff C:\WINDOWS\system32\PSAPI.DLL
              77e00000-77e20fff C:\WINDOWS\system32\NTMARTA.DLL
              76f10000-76f3dfff C:\WINDOWS\system32\WLDAP32.dll
              00b20000-00b2efff C:\WINDOWS\system32\SAMLIB.dll
              77670000-777a8fff C:\WINDOWS\system32\ole32.dll
              6d3f0000-6d424fff E:\bea92\JROCKI~1\jre\bin\jdwp.dll
              6d370000-6d38cfff E:\bea92\JROCKI~1\jre\bin\java.dll
              6d680000-6d68bfff E:\bea92\JROCKI~1\jre\bin\verify.dll
              6d2f0000-6d2f7fff E:\bea92\JROCKI~1\jre\bin\hpi.dll
              6d290000-6d296fff E:\bea92\JROCKI~1\jre\bin\dt_socket.dll
              71b20000-71b60fff C:\WINDOWS\system32\mswsock.dll
              5f270000-5f2c9fff C:\WINDOWS\system32\hnetcfg.dll
              71ae0000-71ae7fff C:\WINDOWS\System32\wshtcpip.dll
              6d530000-6d542fff E:\bea92\jrockit90_150_04\jre\bin\net.dll
              76ed0000-76ef9fff C:\WINDOWS\system32\DNSAPI.dll
              76f70000-76f76fff C:\WINDOWS\System32\winrnr.dll
              76f80000-76f84fff C:\WINDOWS\system32\rasadhlp.dll
              68000000-68034fff C:\WINDOWS\system32\rsaenh.dll
              7c8d0000-7d0cefff C:\WINDOWS\system32\SHELL32.dll
              7d180000-7d1d1fff C:\WINDOWS\system32\SHLWAPI.dll
              77420000-77522fff C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05FDF087\comctl32.dll
              417f0000-417f8fff E:\bea92\jrockit90_150_04\jre\bin\nio.dll
              41690000-4169dfff E:\bea92\weblogic92\server\native\win\32\wlfileio2.dll
              41fd0000-41fd4fff E:\bea92\weblogic92\server\native\win\32\wlntio.dll
              6d660000-6d665fff E:\bea92\jrockit90_150_04\jre\bin\rmi.dll
              72590000-725dbfff C:\WINDOWS\system32\pdh.dll
              762b0000-762f8fff C:\WINDOWS\system32\comdlg32.dll
              44f90000-45026fff C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.4770_x-ww_A689AB02\COMCTL32.dll
              77d00000-77d8afff C:\WINDOWS\system32\OLEAUT32.dll
              48890000-488ccfff C:\WINDOWS\system32\ODBC32.dll
              60950000-60955fff C:\WINDOWS\system32\odbcbcp.dll
              77b90000-77b97fff C:\WINDOWS\system32\VERSION.dll
              761b0000-76242fff C:\WINDOWS\system32\CRYPT32.dll
              76190000-761a1fff C:\WINDOWS\system32\MSASN1.dll
              426f0000-42706fff C:\WINDOWS\system32\odbcint.dll
              5e2e0000-5e2ecfff C:\WINDOWS\system32\perfproc.dll
              5e2f0000-5e2f9fff C:\WINDOWS\system32\perfos.dll
              6d070000-6d1d6fff E:\bea92\jrockit90_150_04\jre\bin\awt.dll
              73070000-73096fff C:\WINDOWS\system32\WINSPOOL.DRV
              73860000-738aafff C:\WINDOWS\system32\ddraw.dll
              73b30000-73b35fff C:\WINDOWS\system32\DCIMAN32.dll
              4db60000-4db9cfff E:\bea92\jrockit90_150_04\jre\bin\fontmanager.dll
              6d230000-6d252fff E:\bea92\jrockit90_150_04\jre\bin\dcpr.dll
              49590000-495befff E:\bea92\jrockit90_150_04\jre\bin\cmm.dll
              4dba0000-4dbbefff E:\bea92\jrockit90_150_04\jre\bin\jpeg.dll
              02800000-028c0fff E:\bea92\JROCKI~1\jre\bin\dbghelp.dll


              "[ACTIVE] ExecuteThread: '28' fo" id=9159 idx=0xa6 tid=2012 lastJavaFrame=0x5E29F764

              Stack 0: start=0x5E260000, end=0x5E2A0000, guards=0x5E263000 (ok), forbidden=0x5E261000
              Thread Stack Trace:
              at _cmMustRemoveReference+1056()@0x008485C0
              -- Java stack --
              at jrockit/vm/Reflect$IClass.runClinit(Ljava/lang/Class;)V(Native Method)
              at jrockit/vm/Reflect$IClass.ensureInitialized(Ljava/lang/Class;)V(Unknown Source)
              at jrockit/reflect/InitialMethodInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
                  at java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)
                  at weblogic/servlet/jsp/JspStub.isJSPClassStale(Ljava/lang/Class;Lweblogic/servlet/jsp/StaleChecker;)Z(JspStub.java:272)
                  at weblogic/servlet/jsp/JspStub.prepareServlet(Lweblogic/servlet/internal/RequestCallback;Z)V(JspStub.java:190)
                  at weblogic/servlet/jsp/JspStub.prepareServlet(Lweblogic/servlet/internal/RequestCallback;)V(JspStub.java:164)
                  at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:235)
                  at weblogic/servlet/internal/ServletStubImpl.onAddToMapException(Ljava/lang/Throwable;Lweblogic/servlet/internal/ServletRequestImpl;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:391)
                  at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:309)
                  at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(ServletStubImpl.java:175)
                  at weblogic/servlet/internal/RequestDispatcherImpl.invokeServlet(ZLjavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/ServletResponseImpl;I)V(RequestDispatcherImpl.java:496)
                  at weblogic/servlet/internal/RequestDispatcherImpl.forward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(RequestDispatcherImpl.java:245)
                  at org/apache/struts/action/RequestProcessor.doForward(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(RequestProcessor.java:1069)
                  at org/apache/struts/action/RequestProcessor.processForwardConfig(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/apache/struts/config/ForwardConfig;)V(RequestProcessor.java:455)
                  at org/apache/struts/action/RequestProcessor.process(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(RequestProcessor.java:279)
                  at org/apache/struts/action/ActionServlet.process(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(ActionServlet.java:1482)
                  at com/dbo2/pipeline/web/PipelineActionServlet.process(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(PipelineActionServlet.java:110)
                  at org/apache/struts/action/ActionServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(ActionServlet.java:507)
                  at javax/servlet/http/HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(HttpServlet.java:743)
                  at javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:856)
                  at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run()Ljava/lang/Object;(StubSecurityHelper.java:225)
                  at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/http/HttpServletRequest;Lweblogic/servlet/internal/ServletRequestImpl;Ljavax/servlet/ServletResponse;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/Servlet;)Ljava/lang/Throwable;(StubSecurityHelper.java:127)
                  at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:283)
                  at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(ServletStubImpl.java:175)
                  at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(WebAppServletContext.java:3211)
                  at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)
                  at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)
                  at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(WebAppServletContext.java:1983)
                  at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(WebAppServletContext.java:1890)
                  at weblogic/servlet/internal/ServletRequestImpl.run()V(ServletRequestImpl.java:1344)
                  at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)
                  at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)
                  at jrockit/vm/RNI.c2java(IIII)V(Native Method)
                  -- end of trace

              Additional information is available in:
                E:\bea92\USER_P~1\domains\DBO2DO~1\jrockit.5544.dump
                E:\bea92\USER_P~1\domains\DBO2DO~1\jrockit.5544.mdmp
              If you see this dump, please open a support case with BEA and
              supply as much information as you can on your system setup and
              the program you were running. You can also search for solutions
              to your problem at http://forums.bea.com in
              the forum jrockit.developer.interest.general.
              ===== END DUMP ===============================================================                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
              • 4. Re: Weblogic Crashing (Jayashankar- OpSource)
                Akshay Koul
                Would suggest you to open a support ticket with Oracle to get the core dump file analyzed.

                However, you can also try using http://download.oracle.com/docs/cd/E19422-01/819-3683/ to dig deeper.

                -Akshay