2 Replies Latest reply: May 17, 2013 12:59 AM by weiishann RSS

    [STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default self-tuning

    weiishann
      Hi all

      This is the default weblogic server that comes installed with the Oracle Enterprise Manager 12c. Everyday at about 12.10am, I will hit this error below.

      +Description
      -----------
      Incident detected using watch rule "StuckThread":
      Watch time: May 15, 2013 12:10:06 AM SGT
      Watch ServerName: EMGC_OMS1
      Watch RuleType: Log
      Watch Rule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
      Watch DomainName: GCDomain
      Watch Data:
      DATE : May 15, 2013 12:10:06 AM SGT
      SERVER : EMGC_OMS1
      MESSAGE : [STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@6f6f578e[
      POST /em/websvcs/emws/ConsoleJobStepExecutorService HTTP/1.1
      Connection: TE
      TE: trailers, deflate, gzip, compress
      User-Agent: Oracle HTTPClient Version 10h
      SOAPAction: ""
      Accept-Encoding: gzip, x-gzip, compress, x-compress
      ECID-Context: 1.0000JuKPI2DDKe85zjL6iW1HZazT000002;lbEIemnE
      Content-type: text/xml; charset=UTF-8
      Content-Length: 1008

      ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      java.net.SocketInputStream.socketRead0(Native Method)
      java.net.SocketInputStream.read(SocketInputStream.java:129)
      oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:730)
      oracle.net.ns.Packet.receive(Packet.java:302)
      oracle.net.ns.DataPacket.receive(DataPacket.java:108)
      oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
      oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
      oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
      oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
      oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1178)
      oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:314)
      oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:205)
      oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:548)
      oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:217)
      oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1115)
      oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1488)
      oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3769)
      oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3904)
      oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1512)
      oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.executeUpdate(PreparedStatementWrapper.java:434)
      oracle.sysman.emmos.impl.RFMPlugIn_Certifications.executeUpdate(RFMPlugIn_Certifications.java:672)
      oracle.sysman.emmos.impl.RFMPlugIn_Certifications.loadCertification(RFMPlugIn_Certifications.java:1147)
      oracle.sysman.emmos.impl.RFMPlugIn_Certifications.loadData(RFMPlugIn_Certifications.java:643)
      oracle.sysman.emmos.impl.RFMPlugIn_Certifications.loadData(RFMPlugIn_Certifications.java:426)
      oracle.sysman.emmos.impl.RFMPlugIn_Certifications.loadData(RFMPlugIn_Certifications.java:379)
      oracle.sysman.emmos.impl.MOS_RFMPlugIn.loadData(MOS_RFMPlugIn.java:227)
      oracle.sysman.emmos.impl.MOS_RFMPlugIn.loadData(MOS_RFMPlugIn.java:183)
      oracle.sysman.emdrep.jobs.commands.UpdateARUTables.loadARUXMLs(UpdateARUTables.java:472)
      oracle.sysman.emdrep.jobs.commands.UpdateARUTables.loadARUXMLs(UpdateARUTables.java:434)
      oracle.sysman.emdrep.jobs.commands.UpdateARUTables.executeCommand(UpdateARUTables.java:323)
      oracle.sysman.emdrep.jobs.CommandStepExecutor.execute(CommandStepExecutor.java:154)
      oracle.sysman.emWebservices.inbound.jobs.worker.ConsoleJobStepExecutor.execute(ConsoleJobStepExecutor.java:321)
      oracle.sysman.emWebservices.inbound.jobs.worker.ConsoleJobStepExecutor.executeStep(ConsoleJobStepExecutor.java:187)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      oracle.j2ee.ws.server.jaxws.ServiceEndpointRuntime.processMessage(ServiceEndpointRuntime.java:370)
      oracle.j2ee.ws.server.jaxws.ServiceEndpointRuntime.processMessage(ServiceEndpointRuntime.java:202)
      oracle.j2ee.ws.server.jaxws.JAXWSRuntimeDelegate.processMessage(JAXWSRuntimeDelegate.java:474)
      oracle.j2ee.ws.server.provider.ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:1187)
      oracle.j2ee.ws.server.WebServiceProcessor$1.run(WebServiceProcessor.java:1120)
      oracle.j2ee.ws.server.WebServiceProcessor$1.run(WebServiceProcessor.java:1118)
      java.security.AccessController.doPrivileged(Native Method)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
      oracle.security.jps.internal.jaas.AccActionExecutor.execute(AccActionExecutor.java:74)
      oracle.security.jps.internal.jaas.CascadeActionExecutor$SubjectPrivilegedExceptionAction.run(CascadeActionExecutor.java:83)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
      weblogic.security.Security.runAs(Security.java:61)
      oracle.security.jps.wls.jaas.WlsActionExecutor.execute(WlsActionExecutor.java:51)
      oracle.security.jps.internal.jaas.CascadeActionExecutor.execute(CascadeActionExecutor.java:56)
      oracle.j2ee.ws.server.provider.GenericProviderPlatform.runAs(GenericProviderPlatform.java:398)
      oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:1138)
      oracle.j2ee.ws.server.provider.ProviderProcessor.doRequestProcessing(ProviderProcessor.java:581)
      oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:233)
      oracle.j2ee.ws.server.WebServiceProcessor.doService(WebServiceProcessor.java:193)
      oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:485)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
      weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:335)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.sysman.emas.fwk.MASConnectionFilter.doFilter(MASConnectionFilter.java:41)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:179)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:550)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adfinternal.view.faces.caching.filter.AdfFacesCachingFilter.doFilter(AdfFacesCachingFilter.java:137)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
      java.security.AccessController.doPrivileged(Native Method)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
      oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

      SUBSYSTEM : WebLogicServer
      USERID : <WLS Kernel>
      SEVERITY : Error
      THREAD : [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'
      MSGID : BEA-000337+

      I am not very familiar with weblogic and will appreciate any help with it. I know that it's stuck because the thread has worked for more than 600s. What what is the reason why it worked more than 600s?

      Thanks!!