1 Reply Latest reply: Apr 10, 2013 9:18 AM by the_assface RSS

    Stuck Threads

    the_assface
      We have an application that customers can load their pdfs up with and we occasionally get stuck threads on a few times a week. Any ideas on where to start looking in order to resolve this long term? Here is the latest log section....thanks.

      ####<Apr 10, 2013 8:05:09 AM CDT> <Error> <WebLogicServer> <FCPORTAL2> <CommandNet2> <[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1365599109210> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,233" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@82aa46d[
      POST /acn/rehost/fdm/submitDocument.do HTTP/1.1
      Cache-Control: no-cache
      Content-Type: multipart/form-data; boundary=---------------------------7dd30123607a2
      Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
      Accept-Encoding: gzip, deflate
      Accept-Language: en-US
      Cookie:
      ]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
      Thread-235 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      weblogic.jdbc.db2.drda.DRDAByteOrderedDataReader.receive(Unknown Source)
      weblogic.jdbc.db2.drda.DRDARequest.submitRequest(Unknown Source)
      weblogic.jdbc.db2.DB2ImplResultSet.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.getAutoGeneratedKeyResultSet(Unknown Source)
      weblogic.jdbc.db2base.BaseStatement.getGeneratedKeys(Unknown Source)
      weblogic.jdbc.wrapper.PreparedStatement_weblogic_jdbc_db2base_BasePreparedStatement40.getGeneratedKeys(Unknown Source)
      sun.reflect.GeneratedMethodAccessor55722.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGeneratedKeysHelper.java:59)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:1731)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:2166)
      org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:24)
      org.hibernate.engine.ActionQueue.execute(ActionQueue.java:230)
      org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:190)
      org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:132)
      org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:83)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:184)
      org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:32)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:159)
      org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:22)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:45)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:480)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:476)
      com.fc.hibernate.DataContext.sessionSave(DataContext.java:71)
      com.fc.commandnet.fdm.SubmitDocumentAction.onSave(SubmitDocumentAction.java:66)
      com.fc.commandnet.fdm.FdmBaseAction.execute(FdmBaseAction.java:94)
      org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
      org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
      org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
      org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
      }

      >

      POST /acn/rehost/fdm/submitDocument.do HTTP/1.1


      ]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
      Thread-235 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      weblogic.jdbc.db2.drda.DRDAByteOrderedDataReader.receive(Unknown Source)
      weblogic.jdbc.db2.drda.DRDARequest.submitRequest(Unknown Source)
      weblogic.jdbc.db2.DB2ImplResultSet.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.getAutoGeneratedKeyResultSet(Unknown Source)
      weblogic.jdbc.db2base.BaseStatement.getGeneratedKeys(Unknown Source)
      weblogic.jdbc.wrapper.PreparedStatement_weblogic_jdbc_db2base_BasePreparedStatement40.getGeneratedKeys(Unknown Source)
      sun.reflect.GeneratedMethodAccessor55722.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGeneratedKeysHelper.java:59)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:1731)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:2166)
      org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:24)
      org.hibernate.engine.ActionQueue.execute(ActionQueue.java:230)
      org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:190)
      org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:132)
      org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:83)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:184)
      org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:32)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:159)
      org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:22)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:45)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:480)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:476)
      com.fc.hibernate.DataContext.sessionSave(DataContext.java:71)
      com.fc.commandnet.fdm.SubmitDocumentAction.onSave(SubmitDocumentAction.java:66)
      com.fc.commandnet.fdm.FdmBaseAction.execute(FdmBaseAction.java:94)
      org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
      org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
      org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
      org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
      }

      SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = FCPORTAL2 TXID = CONTEXTID = TIMESTAMP = 1365599109210
      WatchAlarmType: AutomaticReset
      WatchAlarmResetPeriod: 120000
      >

      ####<Apr 10, 2013 8:07:09 AM CDT> <Error> <WebLogicServer> <FCPORTAL2> <CommandNet2> <[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1365599229217> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,353" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@82aa46d[
      POST /acn/rehost/fdm/submitDocument.do HTTP/1.1
      X-REQUEST-URI: /acn/rehost/fdm/submitDocument.do

      ]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
      Thread-235 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      weblogic.jdbc.db2.drda.DRDAByteOrderedDataReader.receive(Unknown Source)
      weblogic.jdbc.db2.drda.DRDARequest.submitRequest(Unknown Source)
      weblogic.jdbc.db2.DB2ImplResultSet.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.getAutoGeneratedKeyResultSet(Unknown Source)
      weblogic.jdbc.db2base.BaseStatement.getGeneratedKeys(Unknown Source)
      weblogic.jdbc.wrapper.PreparedStatement_weblogic_jdbc_db2base_BasePreparedStatement40.getGeneratedKeys(Unknown Source)
      sun.reflect.GeneratedMethodAccessor55722.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGeneratedKeysHelper.java:59)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:1731)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:2166)
      org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:24)
      org.hibernate.engine.ActionQueue.execute(ActionQueue.java:230)
      org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:190)
      org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:132)
      org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:83)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:184)
      org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:32)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:159)
      org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:22)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:45)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:480)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:476)
      com.fc.hibernate.DataContext.sessionSave(DataContext.java:71)
      com.fc.commandnet.fdm.SubmitDocumentAction.onSave(SubmitDocumentAction.java:66)
      com.fc.commandnet.fdm.FdmBaseAction.execute(FdmBaseAction.java:94)
      org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
      org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
      org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
      org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
      }

      ####<Apr 10, 2013 8:23:26 AM CDT> <Notice> <Stdout> <FCPORTAL2> <CommandNet2> <Unsafe Secondary Finalizer> <<WLS Kernel>> <> <> <1365600206713> <BEA-000000> <log4j: Finalizing appender named [FJ].>
      ####<Apr 10, 2013 8:23:27 AM CDT> <Notice> <Stdout> <FCPORTAL2> <CommandNet2> <[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1365600207399> <BEA-000000> <DBUtilGeneric.executeQuery() exception:
      Connection string: jdbc/OracleMRS

      at java/lang/Shutdown.exit(Shutdown.java:178)
      ^-- Holding lock: java/lang/Class@0x00000000105D0248[biased lock]
      at java/lang/Runtime.exit(Runtime.java:90)
      at java/lang/System.exit(System.java:906)
      at weblogic/Server.main(Server.java:87)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace


      "Finalizer" id=7 idx=0x3c tid=3640 prio=8 alive, in native, native_waiting, daemon
      at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
          at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
          at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
          at java/lang/Thread.run(Thread.java:619)
          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
          -- end of trace

      "Reference Handler" id=8 idx=0x40 tid=3644 prio=10 alive, in native, native_waiting, daemon
          at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
          at java/lang/ref/Reference.access$100(Reference.java:11)
          at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
          -- end of trace

      "(Sensor Event Thread)" id=9 idx=0x44 tid=3648 prio=5 alive, in native, daemon

      "Timer-0" id=12 idx=0x48 tid=3976 prio=5 alive, in native, waiting, daemon
          -- Waiting for notification on: java/util/TaskQueue@0x0000000013C69668[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)[optimized]
      at java/lang/Object.wait(Object.java:485)
      at java/util/TimerThread.mainLoop(Timer.java:483)
      ^-- Lock released while waiting: java/util/TaskQueue@0x0000000013C69668[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "Timer-1" id=13 idx=0x4c tid=4236 prio=5 alive, in native, waiting, daemon
      -- Waiting for notification on: java/util/TaskQueue@0x0000000011574308[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)[optimized]
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0x0000000011574308[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=14 idx=0x50 tid=4240 prio=5 alive, in native, waiting, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0x0000000011574AF8[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at jrockit/vm/Locks.wait(Locks.java:2251)[inlined]
      at java/lang/Object.wait(Object.java:485)[inlined]
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)[optimized]
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0000000011574AF8[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "weblogic.time.TimeEventGenerator" id=15 idx=0x54 tid=4244 prio=9 alive, in native, waiting, daemon
      -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x00000000139B8250[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at jrockit/vm/Locks.wait(Locks.java:2251)[inlined]
      at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)[optimized]
      ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x00000000139B8250[fat lock]
      at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
      at java/lang/Thread.run(Thread.java:619)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "JMAPI event thread" id=16 idx=0x58 tid=4248 prio=5 alive, in native, native_waiting, daemon

      Method)
      -- end of trace