This discussion is archived
4 Replies Latest reply: Jan 29, 2013 6:01 AM by the_assface RSS

Stuck Thread -

the_assface Explorer
Currently Being Moderated
We have an app that has a feature where an advisor can upload pdfs for the client to pick up on the client app. The upload basically stores it on an iSeries for the client pickup. Most os the time this works fine and we believe it always stores the file for client pickup. However, once or twice a week we get a stuck thread on this process. Details from the logs are pasted below. Anyone have recommendations on how best to start troubleshooting this? Thanks...let me know if there is a more appropriate forum for this one.

####<Dec 5, 2012 8:47:38 AM CST> <Error> <WebLogicServer> <FCPORTAL2> <CommandNet2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1354718858020> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,289" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2744144[
POST /acn/rehost/fdm/submitDocument.do HTTP/1.1
Cache-Control: no-cache
Content-Type: multipart/form-data; boundary=---------------------------7dc307a3055a
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US
Cookie: SMIDENTITY=xbnPLVU6TdPbxEkN34O+Ypt+aQLjXjJcRDcwn+HM+lF6RJHtPqsR4b1jT3vKAPSDdVv/NilS19AATC/Uv5svJC0m//a3EsY9tkgUAfKWWWzf8Lmp36xQVbRuEanEZzPyKegHNx26kUop5nT1LyUkQaaTFZStl/WE2md8XUf9oZNYoM9Aydg3LgHuP2ghGBXBSfg0c/l4gYsm/mzhMR0qj1xOe/1q6BZ/mHp+oWdCrtu4ajfmG8v81/f1HWSpf789IUvjxZUDjERzdD1g60Jo9RQcJezkV9qnTQKIvOtnfuxwzoecNX9kTRze0j9Alhdg1d5I2nYpJuCPZyvraSoDK7u52rhyy940e6L8JZUFLWgnCih+4ukx90z5P4N8SI3ZKXbL1xFzQRbUwod/xDw0daGNJZc2FNhpKcLRqkEWeL1KxtYanpX0SyxUsir8luMJA+fP3dYn9A8f8+9LE04xo8aD9Gzl3cGIYpfPKbneQ4jk//a245OkyU+ttw0kG7ez8/O6WpxOFGvuRJ3uG23/ngGhABGl2KM9xvuGRosBMD6pfl1cImC+Uv0DnRh7XtfS7Ad4WWIwy7+2Y0bsji1LmJ1PwusMfFYWy/Q0fLeGAd52GhMPB3+Bp8EISYhgrfnlSxdcbNApbGk9p+WsKNkauJ3jrsMbXTcKrFpm+s6zlTls9L4GHAfxAownkpGMtkcLm6EqIw1cQH0Ge2JZ5AK6nDuBsnQto9Kbw+dszmYPbGn+AQsQG2pM62KX8kw5uQ6+BHFznhET104ZU66U4ZR0IqYVjSZFG54vWrjq2XwjmmLeTlgCiYPfsOKUdntaTk+BnsTNltrQ4ojkO1xkriTkXeJGkdkuZH22; __utma=1.1980564033.1354715806.1354715806.1354715806.1; __utmb=1.3.10.1354715806; __utmz=1.1354715806.1.1.utmcsr=www2.firstcommand.com|utmccn=(referral)|utmcmd=referral|utmcct=/siteminderagent/Forms/DynamicRetry.fcc; SMSESSION=QpQXpDbli6BVjlXE/pvnqLpHedj9rew62O48T3JpWj0A4LFp5fqqA8uC3pIJoLZeln7p3mepjFGnlc5cPcGn9SyXCJMZ0FxDYjJRsKivZYWRvAauslXBjMCpo48FeedZARYzDL0L9dhLXAIop0bD8Gr0xbG/URW1QcBJ9yKAuMBiArFKnaLYaXlPPO185XB5w86d1y9Ov+ULVe/3W+dOTAMkggzforRsG8M4ImMM7g+mh9GMOcmv/oJq493AvkIEMPYZxmBmqVr7PZd3Uh1zPQ55TfmxJv9wxiaNJlYOOorTQrSU4WT+ia70Q5jK0BJB7WTDdrypK9/uBBtGLtJMeNWRELKgPU2QLkUzPBPRIYZIQxuSjA2QnFSXIHtxBLpaNgA+BjLobyYKsV68Jk4PH1sMNY1S5+EVBEGojVYji8DzehAo+4bCTz9qJm4MNgXLQNPsyuPNPtTtskYH6p94RCE5xh/VKfPwd6Lv7/5hpuVwgwnu2Ymkafn4gwSfxd/ewuQ0RbLDi6PFHTbBfaYPB5fJgP+eMZLl4BEKAWJ10lCWGCD0I3bWwUkQqR+S5Ke+aUHhMYYjPd6Y20AfXSQ42PXWjTfx9BqXlg1tLv3+/mWBpRCFLmnx+bEeRNeisdgZf0Ez96F8/T/nZVt27o/2iqF/3eoh9ru1I1RZnDOcJxd0uY0DauHUWpg+1gexQYEIkRiJKi3P22Q9AzhaM0SVFuuqZcO4Jgp1+VEay7muW1OKwNOFeR3+grBqnudb/0p7mAJSABO9qBQIY8Ps5Lfr+ksj79dfYpDn8dA8JD+GdtvGfHF8ih9ddgBzbKrxx8uSMpXWXUNmH0ncRiVUQnMhGzH/PDPTjmSL/kASIEROcsswouV1xn370abdqWAv/qin678N0gWN09W4ne/UtjnuFybGNLyX+F1SKHKOtpRsl5JAuqltt4tZpFgImdNerjdAH6cX1Bwy1rTCJMmn97wWqJUMxDaMmG4mAueiyUg9rdeUpEE2I9f7BKMUv6i4nKvRUUHmZTsJoaUwSARLNhc8e/X/Mti+n7r2aDg/o0CDf86rdcJrPgaCcEmWgP0y+Ek21fr4QVQcvcVJQc7leS8rEQcKoygCB9Pmqt4S+9s+QoOTZ5nNJNyomyW2hice981fFysfnIn3pe0jOQA+z7b3zylvqV0K9frxG2J+6PkgBgnhda9oXkQfQP9stbm93GAAL5jOxCyDDsCLbCYO0zXGCjxoJuCGS05T; __utmc=1; s_cc=true; s_sq=fcfcomcommandnet%3D%2526pid%253DCFP%252520Family%2526pidt%253D1%2526oid%253Dhttps%25253A//www2.firstcommand.com/acn/cms/wwc/CFP%25253FclientID%25253D1300034%252526hideNavs%25253Dtrue%252523%2526ot%253DA%2526oi%253D64; TS44dad5=0483aac1f0beba76487e1b0b933fa5b88541942266a10c7550bf58d96c7f7b9ca28dfd266c7f7b9cfffffffffb626662fc0d6f9a; ASPSESSIONIDCSDAQCSB=MNLHNDJDJOLKIAGPPGLFCDAK; JSESSIONID=1hvhQ1YGsW8RPlhhvfMR3TDC2qyzncNWTqH2LCL8pcyBXv7nP8BM!616071367
Referer: https://www2.firstcommand.com/acn/rehost/fdm/loadDocument.do?action=add&index=-1&pkgID=19126&familyID=1098873
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; HPNTDF; InfoPath.3; .NET4.0C; MS-RTC LM 8; AskTbORJ/5.14.1.20007)
X-Forwarded-For: 67.173.249.193,10.5.106.37
X-REQUEST-URI: /acn/rehost/fdm/submitDocument.do
SERVER_PROTOCOL: HTTP/1.1
SM_TRANSACTIONID: 000080fe0000000050ad17f4f4a63477-0ec0-50bf5980-06d0-026652a3
SERVER_PORT: 80
REQUEST_METHOD: POST
SM_SDOMAIN: .xx.com
SM_REALM: Net App Server Realm
SM_REALMOID: 06-bfcdf830-6d58-4787-8ba4-903fe9a8626a
SM_AUTHTYPE: Form
SM_AUTHREASON: 0
SM_AUTHDIROID: 0e-58775bad-5bf5-4996-b3c3-6984c97ec1b6
SM_AUTHDIRNAME: FCFFieldUsers
SM_AUTHDIRSERVER:
SM_AUTHDIRNAMESPACE: LDAP:
SM_USER: deingram
SM_USERDN: CN=
SM_SERVERSESSIONID: RYCMpT2I7oPfGakPB1Zmvn935qk=
SM_SERVERSESSIONSPEC: RxuXdRr7U/p6AUpHVi9oClsKGK/8ijY9+KuuNcUSMlFYezAZ4t1FRKaeFQCKodGQrjJhWwNzqsrd9e0tdVjVmbdIoWPtclrDTQTIAXeLnr/4xbZHQp/2YOGvlAYyXtAl1ZIan5xj8BGn5RkQVAeqJDsWdZ1UxAG3AipaodknKUGYOzAfzLEAMQ+slkHUH+dSDxLOtxBi7EcDbj3zEB+/UL2WhAmo07aChyDRxz5XeAse6jwIkXUvWbeWznAFXUsBvw65eA/Xs6Ktox7t0TxwkQ1GtmpNmcqoF2PHojGUbGaGYwKqILuk1ae0yhk/T8TbY0lqegcn8huiR1kMCzFycjmjPG8PLnFJB6ktU3ThwydGDS/OJifR2RxJAWptZejC/p6e+paEs09LiQpu65p7/ppWtT0lBZPthqV5YYX/Yov0haNNmxGR77cbfo+pmRJ9GeZDLjgGpqoo0aiRytrYybhlkaYUjsqp1tm3b1pmkwE=
SM_SERVERIDENTITYSPEC: fss5hW1BUnrpv9DTNb4tn7XAtxRtUO9GONYVB2t9iSRxZm/907tRrhDoqIeY7ND7IOLbvmILWiphqYXyZO19IQCXH4ERrIZFTkRNBV35mB73krjKsrmKTlasnrMfWMFmnpLtBn7a5Hj9IEDq0ra5oIkXILzNYsVy2xF/L5xaKwagGZh1DODyhYpE1YhFKmrUzI2+lHxkdTfV39YgLdw1oDZJ/YowR5qLqtpeVXwZvJO+n91iS9byRs8OOIT5MEQi7RlliKX4Q34lrsgpqknNYCVAUZ68k/YuJPqaun+UBeknOzTZOnRhauEncIF1kHbWQPEcNNOhBlq4IYPRg/JfeX2W3qy5lNOaAyuJUK5Jh6dDIlGJhT8lH8/23oHLp/Xp
WebUserID: Ingram, DeNetta
Max-Forwards: 10
X-Forwarded-Host: www2.
X-Forwarded-Server: 10.5.106.74
Content-Length: 90628
Connection: Keep-Alive
X-WebLogic-Cluster-Hash: Q8D5PPZ2Pbl/ys6YKU7LAPHTdeA
X-WebLogic-Request-ClusterInfo: true

]", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
Thread-2183 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.jdbc.db2.drda.DRDAByteOrderedDataReader.receive(Unknown Source)
weblogic.jdbc.db2.drda.DRDARequest.submitRequest(Unknown Source)
weblogic.jdbc.db2.DB2ImplResultSet.close(Unknown Source)
weblogic.jdbc.db2.DB2ImplStatement.close(Unknown Source)
weblogic.jdbc.db2.DB2ImplStatement.getAutoGeneratedKeyResultSet(Unknown Source)
weblogic.jdbc.db2base.BaseStatement.getGeneratedKeys(Unknown Source)
weblogic.jdbc.wrapper.PreparedStatement_weblogic_jdbc_db2base_BasePreparedStatement40.getGeneratedKeys(Unknown Source)
sun.reflect.GeneratedMethodAccessor1135.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGeneratedKeysHelper.java:59)
org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:1731)
org.hibernate.persister.entity.BasicEntityPersister.insert(BasicEntityPersister.java:2166)
org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:24)
org.hibernate.engine.ActionQueue.execute(ActionQueue.java:230)
org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:190)
org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:132)
org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:83)
org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:184)
org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:32)
org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:159)
org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:22)
org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:45)
org.hibernate.impl.SessionImpl.save(SessionImpl.java:480)
org.hibernate.impl.SessionImpl.save(SessionImpl.java:476)
com.fc.hibernate.DataContext.sessionSave(DataContext.java:71)
com.fc.commandnet.fdm.SubmitDocumentAction.onSave(SubmitDocumentAction.java:62)
com.fc.commandnet.fdm.FdmBaseAction.execute(FdmBaseAction.java:94)
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:217)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1481)
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
  • 1. Re: Stuck Thread -
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Hi,

    Looking @ the stack I dont think this is a big issue but for fine tuning you can check the DB2 DS connectivity and enable tunneling on WLS side.

    Now also check whether Native memory is exists or not and check Heap usage if it less then try to increase Heap.

    this will solve your problem.

    Regards,
    Kal
  • 2. Re: Stuck Thread -
    the_assface Explorer
    Currently Being Moderated
    Yes, we have tunneling enabled on the JVM that hosts this app. Native IO is enabled. We have increased our memory as much as 4x and still have the occasional stuck thread with this issue. Is there any tweaks you are recommending on the datasource side or were you just referencing that for the JDBC target for tunneling?

    Thanks
  • 3. Re: Stuck Thread -
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Hi,

    Tunneling make sure whether porotcol channels are alive so by enabling it will make sure that all socket related things will be taken care.

    Secondly I would say try to enable test connection on reserve on your DS and connection retry frequency second from zero to 20.

    This will improve your Data source activity.

    Regards,
    Kal
  • 4. Re: Stuck Thread -
    the_assface Explorer
    Currently Being Moderated
    I had the test table enabled but frequency was set to 0 (disabled)....set it to 20 and will monitor progress and update this post within the next few days. Thanks.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points