This discussion is archived
0 Replies Latest reply: May 16, 2011 12:58 PM by 722441 RSS

Thread Dump

722441 Newbie
Currently Being Moderated
Hi,

We encounter a out of memory issue on weblogic 8.1 and the server crashed in 1-2 days.

Below is the Thread dump, could somebody analysis what is the root cause the issue?
you can get the file at:http://www.4shared.com/document/DzjOwH9m/threaddump-1604.html


Thanks,


Full thread dump Java HotSpot(TM) Server VM (1.4.2_16-b05 mixed mode):

"J-Integra internal timer thread for Jintegra RemOXIDResolver for: Net={{ 0x7, AJEP-EPARTAPP1}, { 0x7, 10.126.64.18}} Sec={{ 0x9, 0xffff, }, { 0x10, 0xffff, }, { 0xa, 0xffff, }, { 0xe, 0xffff, }, { 0x16, 0xffff, }}" daemon prio=5 tid=0x01ac5668 nid=0x2d3 waiting on condition [0x241bf000..0x241bfc28]
     at java.lang.Thread.sleep(Native Method)
     at com.linar.jintegra.ci.run(Unknown Source)

"J-Integra internal timer thread for Jintegra RemOXIDResolver for: Net={{ 0x7, 10.78.124.166[135]}} Sec={}" daemon prio=5 tid=0x02897db0 nid=0x2d1 waiting on condition [0x241ef000..0x241efc28]
     at java.lang.Thread.sleep(Native Method)
     at com.linar.jintegra.ci.run(Unknown Source)

"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0233afe0 nid=0x122 in Object.wait() [0x2424f000..0x2424fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x5d611de8> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x016c0818 nid=0x121 in Object.wait() [0x242df000..0x242dfc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x5d4481e8> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x022b7e28 nid=0x120 in Object.wait() [0x2427f000..0x2427fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x5d956708> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"Thread-26" prio=5 tid=0x0289c838 nid=0x64 waiting on condition [0x242af000..0x242afc28]
     at java.lang.Thread.sleep(Native Method)
     at com.bluemartini.htmlapp.HTMLSaveRequestsDB.run(HTMLSaveRequestsDB.java:75)

"SSLListenThread.Default" prio=5 tid=0x01a08b48 nid=0x61 runnable [0x2430f000..0x2430fc28]
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
     - locked <0x57a633f8> (a java.net.PlainSocketImpl)
     at java.net.ServerSocket.implAccept(ServerSocket.java:448)
     at javax.net.ssl.impl.SSLServerSocketImpl.accept(Unknown Source)
     at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
     at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)

"ListenThread.Default" prio=5 tid=0x0073c870 nid=0x60 runnable [0x2433f000..0x2433fc28]
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
     - locked <0x574cb118> (a java.net.PlainSocketImpl)
     at java.net.ServerSocket.implAccept(ServerSocket.java:448)
     at java.net.ServerSocket.accept(ServerSocket.java:419)
     at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
     at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
     at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)

"Thread-25" prio=5 tid=0x001c2de0 nid=0x5f in Object.wait() [0x2436f000..0x2436fc28]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x57501260> (a java.util.TaskQueue)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x57501260> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"CacheExpiryHelper" prio=5 tid=0x00ea99a0 nid=0x5e in Object.wait() [0x2439f000..0x2439fc28]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x574c5748> (a java.lang.Object)
     at com.bluemartini.cache.ExpiryHelper.run(ExpiryHelper.java:96)
     - locked <0x574c5748> (a java.lang.Object)

"Thread-23" prio=5 tid=0x00a878e0 nid=0x5c waiting on condition [0x24d6f000..0x24d6fc28]
     at java.lang.Thread.sleep(Native Method)
     at com.bluemartini.htmlapp.HTMLSaveTracking.runDBThreads(HTMLSaveTracking.java:585)
     at com.bluemartini.htmlapp.HTMLSaveTracking.run(HTMLSaveTracking.java:436)

"AssocRecomLoader" prio=5 tid=0x01e2c818 nid=0x5b in Object.wait() [0x2675f000..0x2675fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at com.bluemartini.thirdparty.rules.AssocRecomLoader.run(AssociationRecommender.java:984)
     - locked <0x55806088> (a java.util.LinkedList)
     at java.lang.Thread.run(Thread.java:534)

"Thread-22" prio=5 tid=0x00d41ec0 nid=0x5a waiting on condition [0x267bf000..0x267bfc28]
     at java.lang.Thread.sleep(Native Method)
     at com.bluemartini.server.BMServerThread.run(BMServerThread.java:48)

"Thread-21" daemon prio=5 tid=0x00ae6730 nid=0x58 in Object.wait() [0x24d9f000..0x24d9fc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55806508> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-20" daemon prio=5 tid=0x01040288 nid=0x57 in Object.wait() [0x24dcf000..0x24dcfc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55800010> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-19" daemon prio=5 tid=0x01494ab0 nid=0x56 in Object.wait() [0x24dff000..0x24dffc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55803840> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-18" daemon prio=5 tid=0x017068f8 nid=0x55 in Object.wait() [0x24e2f000..0x24e2fc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55800030> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-17" daemon prio=5 tid=0x01e2d290 nid=0x54 in Object.wait() [0x24e5f000..0x24e5fc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55803860> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-16" daemon prio=5 tid=0x0223d830 nid=0x53 in Object.wait() [0x24e8f000..0x24e8fc28]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55800050> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"BMLogThread" prio=5 tid=0x01c95348 nid=0x52 waiting on condition [0x24ebf000..0x24ebfc28]
     at java.lang.Thread.sleep(Native Method)
     at com.bluemartini.dna.BMLogThread.run(BMLogThread.java:140)

"Thread-12" prio=5 tid=0x00f963e8 nid=0x4f in Object.wait() [0x24eef000..0x24eefc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55800070> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-11" prio=5 tid=0x00ced720 nid=0x4e in Object.wait() [0x24f1f000..0x24f1fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55806528> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-10" prio=5 tid=0x00bf40c8 nid=0x4d in Object.wait() [0x24f4f000..0x24f4fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55800080> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-9" prio=5 tid=0x004d3410 nid=0x4c in Object.wait() [0x24f7f000..0x24f7fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55800090> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-8" prio=5 tid=0x01712088 nid=0x4b in Object.wait() [0x25f9f000..0x25f9fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55803880> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"Thread-7" prio=5 tid=0x018f5758 nid=0x4a in Object.wait() [0x263ef000..0x263efc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x55808408> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x013c7608 nid=0x49 runnable [0x265ff000..0x265ffc28]
     at java.net.PlainDatagramSocketImpl.receive(Native Method)
     at java.net.DatagramSocket.receive(DatagramSocket.java:711)
     - locked <0x3c819820> (a java.net.DatagramPacket)
     - locked <0x55818bc8> (a java.net.MulticastSocket)
     at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:180)
     at weblogic.cluster.MulticastManager.execute(MulticastManager.java:399)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"weblogic.health.CoreHealthMonitor" daemon prio=5 tid=0x017f7408 nid=0x48 waiting on condition [0x2664f000..0x2664fc28]
     at java.lang.Thread.sleep(Native Method)
     at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:163)

"Thread-6" prio=5 tid=0x00499a78 nid=0x47 in Object.wait() [0x2669f000..0x2669fc28]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x55803890> (a java.util.TaskQueue)
     at java.util.TimerThread.mainLoop(Timer.java:429)
     - locked <0x55803890> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"ExecuteThread: '0' for queue: 'Multicast'" daemon prio=5 tid=0x01a6ccf0 nid=0x46 in Object.wait() [0x266cf000..0x266cfc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x55813bb8> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"NodeManagerAgent" prio=5 tid=0x01704848 nid=0x45 runnable [0x266ff000..0x266ffc28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:140)
     at java.io.InputStream.read(InputStream.java:89)
     at com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
     at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
     at com.certicom.tls.record.ReadHandler.read(Unknown Source)
     - locked <0x55819d30> (a com.certicom.tls.record.ReadHandler)
     at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
     at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:408)
     at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:450)
     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:182)
     - locked <0x4d802868> (a java.io.InputStreamReader)
     at java.io.InputStreamReader.read(InputStreamReader.java:167)
     at java.io.BufferedReader.fill(BufferedReader.java:136)
     at java.io.BufferedReader.readLine(BufferedReader.java:299)
     - locked <0x4d802868> (a java.io.InputStreamReader)
     at java.io.BufferedReader.readLine(BufferedReader.java:362)
     at weblogic.nodemanager.internal.NMMessage.receive(NMMessage.java:109)
     at weblogic.nodemanager.wlscontrol.NodeManagerCommandListener.readInputCommand(NodeManagerCommandListener.java:136)
     at weblogic.nodemanager.wlscontrol.NodeManagerCommandListener.run(NodeManagerCommandListener.java:60)

"Thread-5" prio=5 tid=0x015514d8 nid=0x44 in Object.wait() [0x2672f000..0x2672fc28]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x558000a0> (a java.util.TaskQueue)
     at java.lang.Object.wait(Object.java:429)
     at java.util.TimerThread.mainLoop(Timer.java:403)
     - locked <0x558000a0> (a java.util.TaskQueue)
     at java.util.TimerThread.run(Timer.java:382)

"LDAPConnThread-2 ldap://32.69.253.106:7300" daemon prio=5 tid=0x01883940 nid=0x42 runnable [0x2678f000..0x2678fc28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
     - locked <0x558185d8> (a java.io.BufferedInputStream)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
     at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-1 ldap://32.69.253.106:7300" daemon prio=5 tid=0x018835f0 nid=0x40 runnable [0x267ef000..0x267efc28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
     - locked <0x5581a4f8> (a java.io.BufferedInputStream)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
     at java.lang.Thread.run(Thread.java:534)

"LDAPConnThread-0 ldap://32.69.253.106:7300" daemon prio=5 tid=0x0154d6e0 nid=0x3f runnable [0x2681f000..0x2681fc28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
     - locked <0x55818618> (a java.io.BufferedInputStream)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
     at java.lang.Thread.run(Thread.java:534)

"VDE Transaction Processor Thread" prio=2 tid=0x014a1aa0 nid=0x3d in Object.wait() [0x2684f000..0x2684fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
     - locked <0x55820d28> (a com.octetstring.vde.backend.standard.TransactionProcessor)
     at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x013109a0 nid=0x3c in Object.wait() [0x2690f000..0x2690fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x55813c38> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x014da920 nid=0x3b in Object.wait() [0x2693f000..0x2693fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x55813cb8> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x014a38a8 nid=0x3a in Object.wait() [0x2696f000..0x2696fc28]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154)
     - locked <0x55816388> (a weblogic.kernel.ExecuteThread)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00cca560 nid=0x39 runnable [0x26a1f000..0x26a1fc28]
     at weblogic.socket.PosixSocketMuxer.poll(Native Method)
     at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
     - locked <0x5582c110> (a java.lang.String)
     at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x012ba460 nid=0x38 waiting for monitor entry [0x26a4f000..0x26a4fc28]
     at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
     - waiting to lock <0x5582c110> (a java.lang.String)
     at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00c78408 nid=0x37 waiting for monitor entry [0x26a7f000..0x26a7fc28]
     at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
     - waiting to lock <0x5582c110> (a java.lang.String)
     at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points