4 Replies Latest reply: Jan 29, 2013 8:01 AM by the_assface RSS

    Stuck Thread -

    the_assface
      We have an app that has a feature where an advisor can upload pdfs for the client to pick up on the client app. The upload basically stores it on an iSeries for the client pickup. Most os the time this works fine and we believe it always stores the file for client pickup. However, once or twice a week we get a stuck thread on this process. Details from the logs are pasted below. Anyone have recommendations on how best to start troubleshooting this? Thanks...let me know if there is a more appropriate forum for this one.

      ####<Dec 5, 2012 8:47:38 AM CST> <Error> <WebLogicServer> <FCPORTAL2> <CommandNet2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1354718858020> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,289" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2744144[
      POST /acn/rehost/fdm/submitDocument.do HTTP/1.1
      Cache-Control: no-cache
      Content-Type: multipart/form-data; boundary=---------------------------7dc307a3055a
      Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
      Accept-Encoding: gzip, deflate
      Accept-Language: en-US
      Cookie: SMIDENTITY=xbnPLVU6TdPbxEkN34O+Ypt+aQLjXjJcRDcwn+HM+lF6RJHtPqsR4b1jT3vKAPSDdVv/NilS19AATC/Uv5svJC0m//a3EsY9tkgUAfKWWWzf8Lmp36xQVbRuEanEZzPyKegHNx26kUop5nT1LyUkQaaTFZStl/WE2md8XUf9oZNYoM9Aydg3LgHuP2ghGBXBSfg0c/l4gYsm/mzhMR0qj1xOe/1q6BZ/mHp+oWdCrtu4ajfmG8v81/f1HWSpf789IUvjxZUDjERzdD1g60Jo9RQcJezkV9qnTQKIvOtnfuxwzoecNX9kTRze0j9Alhdg1d5I2nYpJuCPZyvraSoDK7u52rhyy940e6L8JZUFLWgnCih+4ukx90z5P4N8SI3ZKXbL1xFzQRbUwod/xDw0daGNJZc2FNhpKcLRqkEWeL1KxtYanpX0SyxUsir8luMJA+fP3dYn9A8f8+9LE04xo8aD9Gzl3cGIYpfPKbneQ4jk//a245OkyU+ttw0kG7ez8/O6WpxOFGvuRJ3uG23/ngGhABGl2KM9xvuGRosBMD6pfl1cImC+Uv0DnRh7XtfS7Ad4WWIwy7+2Y0bsji1LmJ1PwusMfFYWy/Q0fLeGAd52GhMPB3+Bp8EISYhgrfnlSxdcbNApbGk9p+WsKNkauJ3jrsMbXTcKrFpm+s6zlTls9L4GHAfxAownkpGMtkcLm6EqIw1cQH0Ge2JZ5AK6nDuBsnQto9Kbw+dszmYPbGn+AQsQG2pM62KX8kw5uQ6+BHFznhET104ZU66U4ZR0IqYVjSZFG54vWrjq2XwjmmLeTlgCiYPfsOKUdntaTk+BnsTNltrQ4ojkO1xkriTkXeJGkdkuZH22; __utma=1.1980564033.1354715806.1354715806.1354715806.1; __utmb=1.3.10.1354715806; __utmz=1.1354715806.1.1.utmcsr=www2.firstcommand.com|utmccn=(referral)|utmcmd=referral|utmcct=/siteminderagent/Forms/DynamicRetry.fcc; SMSESSION=QpQXpDbli6BVjlXE/pvnqLpHedj9rew62O48T3JpWj0A4LFp5fqqA8uC3pIJoLZeln7p3mepjFGnlc5cPcGn9SyXCJMZ0FxDYjJRsKivZYWRvAauslXBjMCpo48FeedZARYzDL0L9dhLXAIop0bD8Gr0xbG/URW1QcBJ9yKAuMBiArFKnaLYaXlPPO185XB5w86d1y9Ov+ULVe/3W+dOTAMkggzforRsG8M4ImMM7g+mh9GMOcmv/oJq493AvkIEMPYZxmBmqVr7PZd3Uh1zPQ55TfmxJv9wxiaNJlYOOorTQrSU4WT+ia70Q5jK0BJB7WTDdrypK9/uBBtGLtJMeNWRELKgPU2QLkUzPBPRIYZIQxuSjA2QnFSXIHtxBLpaNgA+BjLobyYKsV68Jk4PH1sMNY1S5+EVBEGojVYji8DzehAo+4bCTz9qJm4MNgXLQNPsyuPNPtTtskYH6p94RCE5xh/VKfPwd6Lv7/5hpuVwgwnu2Ymkafn4gwSfxd/ewuQ0RbLDi6PFHTbBfaYPB5fJgP+eMZLl4BEKAWJ10lCWGCD0I3bWwUkQqR+S5Ke+aUHhMYYjPd6Y20AfXSQ42PXWjTfx9BqXlg1tLv3+/mWBpRCFLmnx+bEeRNeisdgZf0Ez96F8/T/nZVt27o/2iqF/3eoh9ru1I1RZnDOcJxd0uY0DauHUWpg+1gexQYEIkRiJKi3P22Q9AzhaM0SVFuuqZcO4Jgp1+VEay7muW1OKwNOFeR3+grBqnudb/0p7mAJSABO9qBQIY8Ps5Lfr+ksj79dfYpDn8dA8JD+GdtvGfHF8ih9ddgBzbKrxx8uSMpXWXUNmH0ncRiVUQnMhGzH/PDPTjmSL/kASIEROcsswouV1xn370abdqWAv/qin678N0gWN09W4ne/UtjnuFybGNLyX+F1SKHKOtpRsl5JAuqltt4tZpFgImdNerjdAH6cX1Bwy1rTCJMmn97wWqJUMxDaMmG4mAueiyUg9rdeUpEE2I9f7BKMUv6i4nKvRUUHmZTsJoaUwSARLNhc8e/X/Mti+n7r2aDg/o0CDf86rdcJrPgaCcEmWgP0y+Ek21fr4QVQcvcVJQc7leS8rEQcKoygCB9Pmqt4S+9s+QoOTZ5nNJNyomyW2hice981fFysfnIn3pe0jOQA+z7b3zylvqV0K9frxG2J+6PkgBgnhda9oXkQfQP9stbm93GAAL5jOxCyDDsCLbCYO0zXGCjxoJuCGS05T; __utmc=1; s_cc=true; s_sq=fcfcomcommandnet%3D%2526pid%253DCFP%252520Family%2526pidt%253D1%2526oid%253Dhttps%25253A//www2.firstcommand.com/acn/cms/wwc/CFP%25253FclientID%25253D1300034%252526hideNavs%25253Dtrue%252523%2526ot%253DA%2526oi%253D64; TS44dad5=0483aac1f0beba76487e1b0b933fa5b88541942266a10c7550bf58d96c7f7b9ca28dfd266c7f7b9cfffffffffb626662fc0d6f9a; ASPSESSIONIDCSDAQCSB=MNLHNDJDJOLKIAGPPGLFCDAK; JSESSIONID=1hvhQ1YGsW8RPlhhvfMR3TDC2qyzncNWTqH2LCL8pcyBXv7nP8BM!616071367
      Referer: https://www2.firstcommand.com/acn/rehost/fdm/loadDocument.do?action=add&index=-1&pkgID=19126&familyID=1098873
      User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; HPNTDF; InfoPath.3; .NET4.0C; MS-RTC LM 8; AskTbORJ/5.14.1.20007)
      X-Forwarded-For: 67.173.249.193,10.5.106.37
      X-REQUEST-URI: /acn/rehost/fdm/submitDocument.do
      SERVER_PROTOCOL: HTTP/1.1
      SM_TRANSACTIONID: 000080fe0000000050ad17f4f4a63477-0ec0-50bf5980-06d0-026652a3
      SERVER_PORT: 80
      REQUEST_METHOD: POST
      SM_SDOMAIN: .xx.com
      SM_REALM: Net App Server Realm
      SM_REALMOID: 06-bfcdf830-6d58-4787-8ba4-903fe9a8626a
      SM_AUTHTYPE: Form
      SM_AUTHREASON: 0
      SM_AUTHDIROID: 0e-58775bad-5bf5-4996-b3c3-6984c97ec1b6
      SM_AUTHDIRNAME: FCFFieldUsers
      SM_AUTHDIRSERVER:
      SM_AUTHDIRNAMESPACE: LDAP:
      SM_USER: deingram
      SM_USERDN: CN=
      SM_SERVERSESSIONID: RYCMpT2I7oPfGakPB1Zmvn935qk=
      SM_SERVERSESSIONSPEC: RxuXdRr7U/p6AUpHVi9oClsKGK/8ijY9+KuuNcUSMlFYezAZ4t1FRKaeFQCKodGQrjJhWwNzqsrd9e0tdVjVmbdIoWPtclrDTQTIAXeLnr/4xbZHQp/2YOGvlAYyXtAl1ZIan5xj8BGn5RkQVAeqJDsWdZ1UxAG3AipaodknKUGYOzAfzLEAMQ+slkHUH+dSDxLOtxBi7EcDbj3zEB+/UL2WhAmo07aChyDRxz5XeAse6jwIkXUvWbeWznAFXUsBvw65eA/Xs6Ktox7t0TxwkQ1GtmpNmcqoF2PHojGUbGaGYwKqILuk1ae0yhk/T8TbY0lqegcn8huiR1kMCzFycjmjPG8PLnFJB6ktU3ThwydGDS/OJifR2RxJAWptZejC/p6e+paEs09LiQpu65p7/ppWtT0lBZPthqV5YYX/Yov0haNNmxGR77cbfo+pmRJ9GeZDLjgGpqoo0aiRytrYybhlkaYUjsqp1tm3b1pmkwE=
      SM_SERVERIDENTITYSPEC: fss5hW1BUnrpv9DTNb4tn7XAtxRtUO9GONYVB2t9iSRxZm/907tRrhDoqIeY7ND7IOLbvmILWiphqYXyZO19IQCXH4ERrIZFTkRNBV35mB73krjKsrmKTlasnrMfWMFmnpLtBn7a5Hj9IEDq0ra5oIkXILzNYsVy2xF/L5xaKwagGZh1DODyhYpE1YhFKmrUzI2+lHxkdTfV39YgLdw1oDZJ/YowR5qLqtpeVXwZvJO+n91iS9byRs8OOIT5MEQi7RlliKX4Q34lrsgpqknNYCVAUZ68k/YuJPqaun+UBeknOzTZOnRhauEncIF1kHbWQPEcNNOhBlq4IYPRg/JfeX2W3qy5lNOaAyuJUK5Jh6dDIlGJhT8lH8/23oHLp/Xp
      WebUserID: Ingram, DeNetta
      Max-Forwards: 10
      X-Forwarded-Host: www2.
      X-Forwarded-Server: 10.5.106.74
      Content-Length: 90628
      Connection: Keep-Alive
      X-WebLogic-Cluster-Hash: Q8D5PPZ2Pbl/ys6YKU7LAPHTdeA
      X-WebLogic-Request-ClusterInfo: true

      ]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
      Thread-2183 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      weblogic.jdbc.db2.drda.DRDAByteOrderedDataReader.receive(Unknown Source)
      weblogic.jdbc.db2.drda.DRDARequest.submitRequest(Unknown Source)
      weblogic.jdbc.db2.DB2ImplResultSet.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.close(Unknown Source)
      weblogic.jdbc.db2.DB2ImplStatement.getAutoGeneratedKeyResultSet(Unknown Source)
      weblogic.jdbc.db2base.BaseStatement.getGeneratedKeys(Unknown Source)
      weblogic.jdbc.wrapper.PreparedStatement_weblogic_jdbc_db2base_BasePreparedStatement40.getGeneratedKeys(Unknown Source)
      sun.reflect.GeneratedMethodAccessor1135.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGeneratedKeysHelper.java:59)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:1731)
      org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:2166)
      org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:24)
      org.hibernate.engine.ActionQueue.execute(ActionQueue.java:230)
      org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:190)
      org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:132)
      org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:83)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:184)
      org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:32)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:159)
      org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:22)
      org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:45)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:480)
      org.hibernate.impl.SessionImpl.save(SessionImpl.java:476)
      com.fc.hibernate.DataContext.sessionSave(DataContext.java:71)
      com.fc.commandnet.fdm.SubmitDocumentAction.onSave(SubmitDocumentAction.java:62)
      com.fc.commandnet.fdm.FdmBaseAction.execute(FdmBaseAction.java:94)
      org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
      org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
      org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
      org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
        • 1. Re: Stuck Thread -
          Kalyan Pasupuleti-Oracle
          Hi,

          Looking @ the stack I dont think this is a big issue but for fine tuning you can check the DB2 DS connectivity and enable tunneling on WLS side.

          Now also check whether Native memory is exists or not and check Heap usage if it less then try to increase Heap.

          this will solve your problem.

          Regards,
          Kal
          • 2. Re: Stuck Thread -
            the_assface
            Yes, we have tunneling enabled on the JVM that hosts this app. Native IO is enabled. We have increased our memory as much as 4x and still have the occasional stuck thread with this issue. Is there any tweaks you are recommending on the datasource side or were you just referencing that for the JDBC target for tunneling?

            Thanks
            • 3. Re: Stuck Thread -
              Kalyan Pasupuleti-Oracle
              Hi,

              Tunneling make sure whether porotcol channels are alive so by enabling it will make sure that all socket related things will be taken care.

              Secondly I would say try to enable test connection on reserve on your DS and connection retry frequency second from zero to 20.

              This will improve your Data source activity.

              Regards,
              Kal
              • 4. Re: Stuck Thread -
                the_assface
                I had the test table enabled but frequency was set to 0 (disabled)....set it to 20 and will monitor progress and update this post within the next few days. Thanks.