2 Replies Latest reply: Mar 20, 2012 1:56 AM by Timo Hahn RSS

    Stuck thread issue

    924823
      My application is suddenly throwing stuck thread issues. It is not pointing to any line of code, so it is very difficult to pin point the cause of it. Is there anyway to troubleshoot this issue? Following is a snippet from server logs:
      "Mar 18, 2012 10:07:22 PM PDT","[STUCK] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for ""600"" seconds working on the request ""Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 600371 ms
      [
      POST /hcmTalent/faces/PerformanceWorkArea?_adf.ctrl-state=vflxklzde_4 HTTP/1.1
      Accept: /
      Accept-Language: en-us
      Referer: https://fusionsystemtest-d-external-hcm.us.oracle.com:10620/hcmTalent/faces/PerformanceWorkArea?_afrLoop=199355096973000&webApp=HomePage&fndHomePageViewId=%2FAtkHomePageWelcome&fnd=%2FWEB-INF%2Foracle%2Fapps%2Fhcm%2Fperformance%2Fdocuments%2Fui%2Fflow%2FWorkerOverviewFlow.xml%23WorkerOverviewFlow%3B%3B%3B%23%7BadfBundle%5B%27oracle.apps.hcm.performance.resource.HcmPerformanceTopGenBundle%27%5D%5B%27Header.PerformanceWorkerOverview%27%5D%7D%3Bfalse%3B256&_afrWindowMode=0&_adf.ctrl-state=vflxklzde_4
      adf-rich-message: true
      Content-Type: application/x-www-form-urlencoded; charset=UTF-8
      adf-ads-page-id: 1
      Accept-Encoding: gzip, deflate
      User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.2; BRI/2)
      Content-Length: 2553
      Cache-Control: no-cache
      Cookie: JSESSIONID=gbHLPm8GZFS543CQkSFD9TQGYXJFZhLnztjnZNfgqSyS8JGtx6YY!330096583!NONE; _WL_AUTHCOOKIE_JSESSIONID=fkr1KrBkfarJv2LuQUmN; s_cc=true; s_sq=%5B%5BB%5D%5D; ORA_UCM_INFO=3~AFE82637E27E4173E040548C2C703F4D~Siddharth~Jain~siddharth.j.jain@oracle.com; s_nr=1330930630255; ORASSO_AUTH_HINT=v1.0~20120319121544; ORA_FUSION_PREFS=v1.0~bG9jYWxlPWVufmRlZmF1bHRMYW5ndWFnZU1hcmtlcj10cnVl; ORA_FND_SESSION_RUP3ST2C=DEFAULT_PILLAR:6p9BWFY/Mc3DHLQ1CyU/h9PppXGjCYod6MCqJj3uNmOdSNCC/FEUXnsx71hHRgHY:1332133010536; ObSSOCookie=KyHgFygoI0lxP1dDa6WGZqo50Y8AhuwqSjlqTuiK3n5QwrDguuNJimtE6g2vW61WLXqAsr2URgUBvMiInAaSC3pissP107%2F0XcfkJ15f%2BwK%2FsP7Tf%2FUBNOQ9Dh11jSIMK88TFsxPYMYQ95uNpKZLZZxusqRg6uKxafIRsxf6M%2FSzlPIpAjZC9e3DNms6PCjJZbg6IBAQXxGYi7PCuI6Elyz6snqvxOK2x1tm3MZjaqjhG4kyLC92zF8XPzvHXEq8Y%2FpWse8HraI28K1h1aDE5GplCyzsO6MwfMhTED7qMqAog48mOsUxYSJP33kD1NNV53SJpt%2FqbfVlyyQCCOQy91pVyJv%2BcmETCCqEtRVc9jzSdR8m7QbY20T%2BX51OrYME; BIGipServerfusionsystemtest-d-external-hcm=1482747914.31785.0000
      IS_SSL: ssl
      OAM_REMOTE_USER: ZHRA-WKR1
      OAM_IDENTITY_DOMAIN: OIMIDStore
      ECID-Context: 1.004it5^lon8Fs1GpIwDCif0001LX000mCc;kXjE1ZDLIPINj8ROhKLOgLQRoVAPXJPPnV8K_IUPjITRXURO_NNRmRPGmLPO
      Connection: Keep-Alive
      X-Forwarded-For: 10.177.138.51
      Proxy-Client-IP: 10.242.1.207
      X-WebLogic-KeepAliveSecs: 30
      X-WebLogic-Request-ClusterInfo: true
      x-weblogic-cluster-hash: RXIPQJi5dtTc/kKQr+GATBGOOQw

      ]"", which is more than the configured time (StuckThreadMaxTime) of ""600"" seconds. Stack trace:
      Thread-162 ""[STUCK] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'"" <alive, suspended, priority=1, DAEMON> {
      java.util.HashMap.entrySet(HashMap.java:1128)
      oracle.jbo.server.ApplicationModuleImpl.passivateInitializedEOs(ApplicationModuleImpl.java:8007)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateState(ApplicationModuleImpl.java:7628)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateAMState(ApplicationModuleImpl.java:7612)
      oracle.jbo.server.Serializer.passivate(Serializer.java:214)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:282)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:267)
      oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5883)
      ^-- Holding lock: oracle.jbo.JboSyncLock@13b6b09a[thin lock]
      oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5878)
      oracle.jbo.server.ApplicationPoolMessageHandler.doPoolMessage(ApplicationPoolMessageHandler.java:117)
      oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:9139)
      ^-- Holding lock: oracle.jbo.JboSyncLock@13b6b09a[thin lock]
      oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:4573)
      oracle.jbo.common.ampool.ApplicationPoolImpl.doManagedCheckin(ApplicationPoolImpl.java:2747)
      oracle.jbo.common.ampool.ApplicationPoolImpl.releaseApplicationModule(ApplicationPoolImpl.java:1558)
      oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:771)
      oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:764)
      oracle.adf.model.bc4j.DCJboDataControl.releaseApplicationModule(DCJboDataControl.java:2413)
      oracle.adf.model.bc4j.DCJboDataControl.endRequest(DCJboDataControl.java:2089)
      ^-- Holding lock: java.lang.Object@13a7e724[thin lock]
      oracle.adf.model.dcframe.DataControlFrameImpl.endRequestInternal(DataControlFrameImpl.java:721)
      oracle.adf.model.dcframe.DataControlFrameImpl.endRequestInternal(DataControlFrameImpl.java:721)
      oracle.adf.model.dcframe.DataControlFrameImpl.endRequestInternal(DataControlFrameImpl.java:721)
      oracle.adf.model.dcframe.DataControlFrameImpl.endRequest(DataControlFrameImpl.java:639)
      ^-- Holding lock: java.util.concurrent.locks.ReentrantLock@12ded2b7[thin lock]
      oracle.adf.model.BindingContext.endRequest(BindingContext.java:2073)
      oracle.adf.model.BindingRequestHandler.invokeEndRequest(BindingRequestHandler.java:397)
      oracle.adf.model.BindingRequestHandler.endRequest(BindingRequestHandler.java:307)
      oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:147)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.adf.view.page.editor.webapp.WebCenterComposerFilter.doFilter(WebCenterComposerFilter.java:80)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:72)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
      oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:231)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
      org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.webcenter.framework.events.dispatcher.EventDispatcherFilter.doFilter(EventDispatcherFilter.java:42)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.webcenter.generalsettings.model.filter.GeneralSettingsLocaleFilter.doFilter(GeneralSettingsLocaleFilter.java:64)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.apps.fnd.applcore.common.ApplSessionFilter.doFilter(ApplSessionFilter.java:123)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:127)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:74)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:147)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:167)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.bpel.services.workflow.client.worklist.util.WorkflowFilter.doFilter(WorkflowFilter.java:120)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
      oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
      }
      ","Error","BEA-000337","","7","","","TalentManagementServer_1","WebLogicServer","","","","slcad950","10.240.96.87","<WLS Kernel>","","[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'","","","","","","","","","",
      "Mar 18, 2012 10:07:35 PM PDT","ADFc: Attempt to acquire root view port request lock failed due to timeout, holding request number = '17', lock was acquired by thread '[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)''.","Error","ADFC-50015","","1","004it5_XL6lFs1GpIwDCif0001LX000mYG","0:1","TalentManagementServer_1","oracle.adfinternal.controller.state.RootViewPortContextImpl","","","","slcad950","10.240.96.87","ZHRA-WKR1","","[ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'","","","","","","","","URI;","/hcmTalent/faces/PerformanceWorkArea;",
      "Mar 18, 2012 10:08:10 PM PDT","You cannot complete this transaction because your person information is incomplete. Contact an HR specialist or other HR administrator. (HRT-975099)","Error","HRT-975099","","1","004it6BP01aFs1GpIwDCif0001LX000ngF","0:1","TalentManagementServer_1","oracle.apps","","","","slcad950","10.240.96.87","ZHRA-WKR1","","22","","","","","","","","URI;","/hcmTalent/faces/PerformanceWorkArea;",
      "Mar 18, 2012 10:08:14 PM PDT","You cannot complete this transaction because your person information is incomplete. Contact an HR specialist or other HR administrator. (HRT-975099)","Error","HRT-975099","","1","004it6BbugWFs1GpIwDCif0001LX000nga","0:1","TalentManagementServer_1","oracle.apps","","","","slcad950","10.240.96.87","ZHRA-WKR1","","17","","","","","","","","URI;","/hcmTalent/faces/PerformanceWorkArea;",