0 Replies Latest reply: Aug 6, 2014 11:06 PM by user7753540 RSS

    Request Help on Analyzing the thread dump

    user7753540

      Hi

       

       

      We are facing an issue with one of our environment having stuck threads. We took some thread dumps and used threadlogic tool to have a look at it. Could you please have a look at this and let me know if we can identify the root cause from this dump.

       

       

      The same code is working on a different server with a similar configuration.

       

       

      Server Out Log

       

      <Aug 6, 2014 7:48:19 AM PDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1dd5b92[

      ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

       

       

       

      ADMIN Server Log

       

       

      ####<Aug 6, 2014 7:48:19 AM PDT> <Error> <WebLogicServer> <b2bfst3> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuni

      ng)'> <<WLS Kernel>> <> <> <1407336499946> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604

      " seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1dd5b92[

      GET /businesscare/change_requests/form_change_plan.jsp?wtRatePlanChangeType=individualplanchange&_requestid=41 HTTP/1.1

      User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0

      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

      Accept-Language: en-US,en;q=0.5

      Accept-Encoding: gzip, deflate

      Referer: http://b2bfsta.edc.cingular.net:8380/businesscare/index.jsp?rpt=true&_requestid=32

      Cookie: cust_type=b2b; mbox=PC#1407196422299-134552.22_11#1408545276|check#true#1407335736|session#1407335538555-615675#1407337536; AMCV_att1=T; s_fid=0435A2

      16F6EF6C7C-00F332FA03AB7740; s_pers=%20s_vnum%3D1409500800724%2526vn%253D6%7C1409500800724%3B%20gpv_v17%3Dpoc%257Cb2bfsta.edc%257Cbusinesscare%7C140733767851

      6%3B%20s_nr%3D1407335878517-Repeat%7C1409927878517%3B%20s_invisit%3Dtrue%7C1407337678517%3B%20s_lv%3D1407335878518%7C1501943878518%3B%20s_lv_s%3DLess%2520tha

      n%25201%2520day%7C1407337678518%3B; PSESSIONID=TzNkThPbgd3RY9!-1955338381; DSESSIONID=52yZTv7XRN5h6yvpBkC7pzykqCnTHvFjhWLM26QnG6GZjl9d31cT!-1758824196; brows

      er=Firefox; s_sess=%20SC_LINKS%3D%3B; s_cc=true; cust_type=b2b; B2B_GTM=B2BBTH

      Connection: Keep-Alive

      WL-Proxy-SSL: false

      WL-Proxy-Client-IP: 170.252.51.6

      Proxy-Client-IP: 170.252.51.6

      X-Forwarded-For: 170.252.51.6

      X-WebLogic-KeepAliveSecs: 30

      X-WebLogic-Force-JVMID: -1758824196

       

       

      ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

      java.util.zip.Inflater.inflateBytes(Native Method)

      java.util.zip.Inflater.inflate(Inflater.java:238)

      java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)

      java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)

      java.io.FilterInputStream.read(FilterInputStream.java:66)

              java.io.DataInputStream.readUnsignedShort(DataInputStream.java:321)

      java.io.DataInputStream.readUTF(DataInputStream.java:572)

      java.io.DataInputStream.readUTF(DataInputStream.java:547)

      com.sun.xml.bind.v2.bytecode.ClassTailor.tailor(ClassTailor.java:142)

              com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.tailor(AccessorInjector.java:119)

      com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:79)

      com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:165)

      com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:261)

      com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.<init>(TransducedAccessor.java:231)

      com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor.get(TransducedAccessor.java:173)

      com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty.<init>(SingleElementLeafProperty.java:83)

              sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)

       

       

       

      Thread Dump Summary is as below.

       

       

      e

      Dump No. 14

       

      Thread Dump Main Thread

      weblogic.Server

      Thread Dump Health

      WARNING   

      Deadlock Found

      false

      Thread Groups

      11

      Total Number of threads

      461

      Number of Locks (with threads blocked)

      1

      Number of blocked threads

      3

      Number of running threads

      44

       

       

      9% of threads are running Healthy (not waiting or blocked).   

       

      Critical Advisories (WATCH, WARNING or FATAL levels) Found

       

      Thread Advisory Name

      Too Many Threads

      Health Level

      WARNING   

      Keyword

      TooManyThreads

      Description

      Using lots of threads - can be a resource drain or lead to too many context switches

      Advice

      Check why there are so many threads.
      Check if this server alone getting all the requests (non-uniform load distribution or loopbacks or peak bursts in traffic).
      Try to throttle the requests.
      Try to reduce max number of threads.
      Use custom WM with Min thread contraints to separate traffic and ensure requests for one application does not overwhelm all other server requests

       

      Thread Advisory Name

      Hot Spots

      Health Level

      WATCH   

      Keyword

      HotCallPattern

      Description

      Multiple Threads executing same code path

      Advice

      Indicates heavily executed code paths. Need not be an indicator of problem but ensure there are no blocking locks or bottlenecks, sufficient resources are available for execution under load.
      Remote service being invoked is responsive and scaling well to handle increased load.
      Use custom Work Manager and dispatch policy to avoid thread starvation or to use dedicated threads for execution

       

      Thread Advisory Name

      JMX Invocation

      Health Level

      WATCH   

      Keyword

      javax.management.remote.rmi

      Description

      JMX invocations

      Advice

      Ensure its not an overhead/drain on performance if multiple threads attempting to query/invoke/get/update MBeans

       

      Thread Advisory Name

      Thread Sleep

      Health Level

      WATCH   

      Keyword

      Thread.sleep

      Description

      Java Thread sleeping

      Advice

      Use wait-notify, instead of repeated sleep-awake-check if possible

       

       

       

       

       

       

       

       

      Please let me know if any of you can advice me on this.