1 Reply Latest reply: Jun 5, 2013 5:01 AM by dimo RSS

    Threads getting stuck in reflection

    dimo
      Hi,

      during our nightly performance tests some weblogic instances start responding very slowly. The reason seems to be in many threads getting stuck doing reflection. The top of the stacktrace looks like this:

      ####<03.06.2013 21:49 Uhr MST> <Error> <WebLogicServer> <XXX> <XXX> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1370321366808> <BEA-000337> <[STUCK] ExecuteThread: '172' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "613" seconds working on the request "XXX", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      Thread-10282 "[STUCK] ExecuteThread: '172' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      java.lang.Class.getDeclaredConstructors0(Class.java:???)
      java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
      java.lang.Class.getConstructor0(Class.java:2699)
      java.lang.Class.newInstance0(Class.java:318)
      java.lang.Class.newInstance(Class.java:305)
      sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
      sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:118)
      sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      org.hibernate.validator.metadata.ConstraintDescriptorImpl.buildAnnotationParameterMap(ConstraintDescriptorImpl.java:318)
      org.hibernate.validator.metadata.ConstraintDescriptorImpl.<init>(ConstraintDescriptorImpl.java:150)
      org.hibernate.validator.metadata.ConstraintDescriptorImpl.<init>(ConstraintDescriptorImpl.java:170)

      It happens on seemingly random instances and causes a lot of other problems afterwards (not enough threads to process new requests, used up jdbc connection pools, etc).

      Weblogic versions:
      WebLogic Server Temporary Patch for 13340309 Thu Feb 16 18:30:21 IST 2012 WebLogic Server Temporary Patch for 13019800 Mon Jan 16 16:53:54 IST 2012 WebLogic Server Temporary Patch for BUG13391585 Thu Feb 02 10:18:36 IST 2012 WebLogic Server Temporary Patch for 13516712 Mon Jan 30 15:09:33 IST 2012 WebLogic Server Temporary Patch for BUG13641115 Tue Jan 31 11:19:13 IST 2012 WebLogic Server Temporary Patch for BUG13603813 Wed Feb 15 19:34:13 IST 2012 WebLogic Server Temporary Patch for 13424251 Mon Jan 30 14:32:34 IST 2012 WebLogic Server Temporary Patch for 13361720 Mon Jan 30 15:24:05 IST 2012 WebLogic Server Temporary Patch for BUG13421471 Wed Feb 01 11:24:18 IST 2012 WebLogic Server Temporary Patch for BUG13657792 Thu Feb 23 12:57:33 IST 2012 WebLogic Server 12.1.1.0 Wed Dec 7 08:40:57 PST 2011 1445491

      JRockit:
      Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)

      Any ideas what is going wrong and more importantly how to fix it?

      Thanks
      Dimo
        • 1. Re: Threads getting stuck in reflection
          dimo
          Seems to be related to the -XgcPrio:pausetime -XpauseTarget:200ms command line options. When I removed them, jrockit uses throughput gc and the problem does not arise. However, I would like to be able to control the gc pause time a bit as we have 96th percentile SLAs and the throughput collector affects more then 4% of the requests....