4 Replies Latest reply: Apr 14, 2004 2:03 PM by 3004 RSS

    Weblogic hangs

    3004
      WL6.0 SP2 on W2K
      Oracle 8.1
      java 1.3.1

      We're running into problems when loadtesting our app. The server
      basically stops responding to requests, including WL pings
      ("weblogic.admin ping") and monitor updates.

      It's a dual-processor W2K box and task manager shows the "java"
      process exactly pegged at 50%... as if there's a single high-priority
      runaway thread that's eating one processor's worth of cycles and
      starving everyone else. Actually I suppose "blocking" would be more
      accurate, since there is spare CPU capacity. When this happens, we
      typically see one CPU hovering around 70-80% and the other 20-30%,
      adding up to exactly 50% for the java process.

      This doesn't appear to be a garbage collection issue, as we've been
      running with verbose GC and pauses are under 5 seconds or so.

      A thread dump is attached. I don't really know how to read this
      stuff, but the places where the various threads are "stuck" seem a bit
      odd:
      18 stuck at java.lang.Class.forName0
      10 stuck at java.lang.reflect.Field.set

      Any help greatly appreciated.

      Full thread dump:

      "Thread-52" daemon prio=5 tid=0x1e881aa0 nid=0x668 waiting on monitor
      [0x207df000..0x207dfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-51" daemon prio=5 tid=0x1f572e80 nid=0x670 runnable
      [0x2079f000..0x2079fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-50" daemon prio=5 tid=0x1f4b6b98 nid=0x644 runnable
      [0x2075f000..0x2075fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-49" daemon prio=5 tid=0x1f4d9f00 nid=0x660 runnable
      [0x2071f000..0x2071fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-48" daemon prio=5 tid=0x1f49f7e8 nid=0x32c runnable
      [0x206df000..0x206dfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-47" daemon prio=5 tid=0x1f6c8f10 nid=0x494 runnable
      [0x2069f000..0x2069fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-46" daemon prio=5 tid=0x1e6681d0 nid=0x16c runnable
      [0x2065f000..0x2065fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-45" daemon prio=5 tid=0x1e8475a8 nid=0x66c runnable
      [0x2061f000..0x2061fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-44" daemon prio=5 tid=0x1e7c55a0 nid=0x124 runnable
      [0x205df000..0x205dfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-43" daemon prio=5 tid=0x1f57afb0 nid=0x528 runnable
      [0x2059f000..0x2059fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-42" daemon prio=5 tid=0x1e853d90 nid=0x350 runnable
      [0x2055f000..0x2055fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-41" daemon prio=5 tid=0x1e826738 nid=0x65c runnable
      [0x2051f000..0x2051fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-40" daemon prio=5 tid=0x1e826438 nid=0x5b8 runnable
      [0x204df000..0x204dfdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-39" daemon prio=5 tid=0x1e8857d0 nid=0x5d4 runnable
      [0x2049f000..0x2049fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-38" daemon prio=5 tid=0x1f6af378 nid=0x5ac runnable
      [0x2045f000..0x2045fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-37" daemon prio=5 tid=0x1f582bd8 nid=0x4a0 runnable
      [0x2041f000..0x2041fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-36" daemon prio=5 tid=0x1f9851d8 nid=0x4c4 runnable
      [0x203df000..0x203dfdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-35" daemon prio=5 tid=0x1e81adb8 nid=0x55c runnable
      [0x2039f000..0x2039fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-34" daemon prio=5 tid=0x1e6b65d8 nid=0x4f8 runnable
      [0x2035f000..0x2035fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-33" daemon prio=5 tid=0x1e7ff608 nid=0x538 runnable
      [0x2031f000..0x2031fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-32" daemon prio=5 tid=0x1e876d08 nid=0x5e8 runnable
      [0x202df000..0x202dfdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-31" daemon prio=5 tid=0x1e818cb0 nid=0x144 runnable
      [0x2029f000..0x2029fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-30" daemon prio=5 tid=0x1f592590 nid=0x52c runnable
      [0x2025f000..0x2025fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-29" daemon prio=5 tid=0x1e7d14f8 nid=0x578 runnable
      [0x2021f000..0x2021fdc4]
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:195)
           at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:45)
           at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:655)
           at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:246)
           at javax.naming.InitialContext.init(InitialContext.java:222)
           at javax.naming.InitialContext.<init>(InitialContext.java:178)
           at com.company.cf.control.common.StatefulEjb.loadDataSource(StatefulEjb.java:168)
           at com.company.cf.control.common.StatefulEjb.getConnection(StatefulEjb.java:130)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjb.isLocked(DomainConfigurationEjb.java:43)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbImpl.isLocked(DomainConfigurationEjbImpl.java:110)
           at com.company.cf.control.domain.ejb.DomainConfigurationEjbEOImpl.isLocked(DomainConfigurationEjbEOImpl.java:30)
           at com.company.cf.control.account.ejb.UserAccountWebImpl$1.run(UserAccountWebImpl.java:89)
           at java.util.TimerThread.mainLoop(Timer.java:435)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-28" daemon prio=5 tid=0x1e843bb0 nid=0x5b0 waiting on monitor
      [0x201df000..0x201dfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-27" daemon prio=5 tid=0x1e7ead98 nid=0x120 waiting on monitor
      [0x2019f000..0x2019fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-26" daemon prio=5 tid=0x1e87ce70 nid=0x580 waiting on monitor
      [0x2015f000..0x2015fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-25" daemon prio=5 tid=0x1e7cbe80 nid=0x514 waiting on monitor
      [0x2011f000..0x2011fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-24" daemon prio=5 tid=0x1e5650f0 nid=0x464 waiting on monitor
      [0x200df000..0x200dfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-23" daemon prio=5 tid=0x1e7b1358 nid=0x258 waiting on monitor
      [0x2009f000..0x2009fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-22" daemon prio=5 tid=0x1e7a1b70 nid=0x474 waiting on monitor
      [0x2005f000..0x2005fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-21" daemon prio=5 tid=0x1e68c6d8 nid=0x44c waiting on monitor
      [0x2001f000..0x2001fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-20" daemon prio=5 tid=0x1f780008 nid=0x29c waiting on monitor
      [0x1ffdf000..0x1ffdfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-19" daemon prio=5 tid=0x1e74e708 nid=0x608 waiting on monitor
      [0x1ff9f000..0x1ff9fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-18" daemon prio=5 tid=0x1e74eb28 nid=0x490 waiting on monitor
      [0x1ff5f000..0x1ff5fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-17" daemon prio=5 tid=0x1e74ede0 nid=0x610 waiting on monitor
      [0x1ff1f000..0x1ff1fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-16" daemon prio=5 tid=0x1e679f40 nid=0x600 waiting on monitor
      [0x1fedf000..0x1fedfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-15" daemon prio=5 tid=0x1e67a028 nid=0x5d0 waiting on monitor
      [0x1fe9f000..0x1fe9fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-14" daemon prio=5 tid=0x1e74e838 nid=0x13c waiting on monitor
      [0x1fe5f000..0x1fe5fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-13" daemon prio=5 tid=0x1e668608 nid=0x5dc waiting on monitor
      [0x1fe1f000..0x1fe1fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-12" daemon prio=5 tid=0x1e74ec10 nid=0x568 waiting on monitor
      [0x1fddf000..0x1fddfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-11" daemon prio=5 tid=0x1e74e920 nid=0x468 waiting on monitor
      [0x1fd9f000..0x1fd9fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-10" daemon prio=5 tid=0x1e6beb00 nid=0x5f4 waiting on monitor
      [0x1fd5f000..0x1fd5fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-9" daemon prio=5 tid=0x1f7d2688 nid=0x4ec waiting on monitor
      [0x1fd1f000..0x1fd1fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-8" daemon prio=5 tid=0x1e858450 nid=0x530 waiting on monitor
      [0x1fcdf000..0x1fcdfdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-7" daemon prio=5 tid=0x1e6b9410 nid=0x5c8 waiting on monitor
      [0x1fc9f000..0x1fc9fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-6" daemon prio=5 tid=0x1e77f570 nid=0x1e4 waiting on monitor
      [0x1f45f000..0x1f45fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-3" daemon prio=5 tid=0x1e63d6c0 nid=0x27c waiting on monitor
      [0x1f39f000..0x1f39fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "Thread-2" daemon prio=5 tid=0x1dca86e8 nid=0x4dc waiting on monitor
      [0x1f35f000..0x1f35fdc4]
           at java.lang.Object.wait(Native Method)
           at java.util.TimerThread.mainLoop(Timer.java:432)
           at java.util.TimerThread.run(Timer.java:385)

      "SSLListenThread" prio=5 tid=0x1e612d28 nid=0x20c runnable
      [0x1f30f000..0x1f30fdc4]
           at java.net.PlainSocketImpl.socketAccept(Native Method)
           at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:413)
           at java.net.ServerSocket.implAccept(ServerSocket.java:241)
           at java.net.ServerSocket.accept(ServerSocket.java:222)
           at weblogic.security.SSL.SSLServerSocket.acceptNoHandshake(SSLServerSocket.java:126)
           at weblogic.security.SSL.SSLServerSocket.accept(SSLServerSocket.java:117)
           at weblogic.t3.srvr.ListenThread.run(ListenThread.java:247)

      "ListenThread" prio=5 tid=0x1e612c40 nid=0x5a0 runnable
      [0x1f2cf000..0x1f2cfdc4]
           at java.net.PlainSocketImpl.socketAccept(Native Method)
           at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:413)
           at java.net.ServerSocket.implAccept(ServerSocket.java:241)
           at java.net.ServerSocket.accept(ServerSocket.java:222)
           at weblogic.t3.srvr.ListenThread.run(ListenThread.java:247)

      "Application Manager Thread" prio=5 tid=0x1e51b430 nid=0x604 runnable
      [0x1f28f000..0x1f28fdc4]
           at java.lang.Class.getMethod0(Native Method)
           at java.lang.Class.getMethod(Class.java:888)
           at weblogic.management.internal.DynamicMBeanImpl.getGetter(DynamicMBeanImpl.java:1343)
           at weblogic.management.internal.DynamicMBeanImpl.getAttribute(DynamicMBeanImpl.java:430)
           at weblogic.management.internal.DynamicMBeanImpl.getAttribute(DynamicMBeanImpl.java:394)
           at weblogic.management.internal.ConfigurationMBeanImpl.getAttribute(ConfigurationMBeanImpl.java:227)
           at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1181)
           at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151)
           at weblogic.management.internal.MBeanProxy.getAttribute(MBeanProxy.java:225)
           at weblogic.management.internal.MBeanProxy.invoke(MBeanProxy.java:158)
           at $Proxy10.isDeployed(Unknown Source)
           at weblogic.management.mbeans.custom.ApplicationManager.poll(ApplicationManager.java:443)
           at weblogic.management.mbeans.custom.ApplicationManager.poll(ApplicationManager.java:380)
           at weblogic.management.mbeans.custom.ApplicationManager.update(ApplicationManager.java:152)
           at weblogic.management.mbeans.custom.ApplicationManager$ApplicationPoller.run(ApplicationManager.java:643)

      "ExecuteThread: '1' for queue: '__weblogic_admin_rmi_queue'" daemon
      prio=5 tid=0x1db54b48 nid=0x59c waiting on monitor
      [0x1f20f000..0x1f20fdc4]
           at java.lang.Object.wait(Native Method)
           at java.lang.Object.wait(Object.java:420)
           at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)

      "ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemon
      prio=5 tid=0x1dbe6b00 nid=0x61c waiting on monitor
      [0x1f1cf000..0x1f1cfdc4]
           at java.lang.Object.wait(Native Method)
           at java.lang.Object.wait(Object.java:420)
           at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)

      "ExecuteThread: '1' for queue: '__weblogic_admin_html_queue'" daemon
      prio=5 tid=0x1dbe6a00 nid=0x560 waiting on monitor
      [0x1f18f000..0x1f18fdc4]
           at java.lang.Object.wait(Native Method)
           at java.lang.Object.wait(Object.java:420)
           at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)

      "ExecuteThread: '0' for queue: '__weblogic_admin_html_queue'" daemon
      prio=5 tid=0x1db4fd90 nid=0x614 waiting on monitor
      [0x1f14f000..0x1f14fdc4]
           at java.lang.Object.wait(Native Method)
           at java.lang.Object.wait(Object.java:420)
           at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)

      "TimeEventGenerator" daemon prio=5 tid=0x1e4b66c8 nid=0x598 runnable
      [0x1f10f000..0x1f10fdc4]
           at java.lang.Object.wait(Native Method)
           at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
           at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:138)
           at java.lang.Thread.run(Thread.java:484)

      "TimeEventGenerator" daemon prio=5 tid=0x1e4b64c8 nid=0x5ec runnable
      [0x1f0cf000..0x1f0cfdc4]
           at java.lang.Object.wait(Native Method)
           at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
           at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:138)
           at java.lang.Thread.run(Thread.java:484)

      "SpinnerRandomSource" daemon prio=5 tid=0x1e4ac6a0 nid=0x620 waiting
      on monitor [0x1f08f000..0x1f08fdc4]
           at java.lang.Object.wait(Native Method)
           at java.lang.Object.wait(Object.java:420)
           at weblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:92)
           at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:109)

      "ExecuteThread: '29' for queue: 'default'" daemon prio=5
      tid=0x1e4a6e40 nid=0x5a8 runnable [0x1f04f000..0x1f04fdc4]
           at weblogic.socket.NTSocketMuxer.getNextSocket(Native Method)
           at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:501)
           at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
           at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

      "ExecuteThread: '28' for queue: 'default'" daemon prio=5
      tid=0x1e4a62a0 nid=0x5fc runnable [0x1f00f000..0x1f00fdc4]
           at weblogic.socket.NTSocketMuxer.getNextSocket(Native Method)
           at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:501)
           at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
           at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

      "ExecuteThread: '27' for queue: 'default'" daemon prio=5
      tid=0x1e4a5700 nid=0x654 runnable [0x1efcf000..0x1efcfdc4]
           at weblogic.socket.NTSocketMuxer.getNextSocket(Native Method)
           at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:50
        • 1. Re: Weblogic hangs
          3004
          First things first: You are (or someone is) spinning off a large number of
          threads. Any explanation on those?

          You have only one free execute thread, which would explain the
          unresponsiveness.
          "ExecuteThread: '5' for queue: 'default'" daemon prio=5 tid=0x1dcc5190
          nid=0x4fc waiting on monitor [0x1ea4f000..0x1ea4fdc4]
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:420)
          at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)
          However, it appears like you are locked up / blocked up on some reflection
          work that you are doing.

          Any more info?

          Peace,

          --
          Cameron Purdy
          Tangosol, Inc.
          Clustering Weblogic? You're either using Coherence, or you should be!
          Download a Tangosol Coherence eval today at http://www.tangosol.com/



          "Fred Schnitzer" <thegap2@excite.com> wrote in message
          news:30545430.0206021333.b64468e@posting.google.com...
          WL6.0 SP2 on W2K
          Oracle 8.1
          java 1.3.1

          We're running into problems when loadtesting our app. The server
          basically stops responding to requests, including WL pings
          ("weblogic.admin ping") and monitor updates.

          It's a dual-processor W2K box and task manager shows the "java"
          process exactly pegged at 50%... as if there's a single high-priority
          runaway thread that's eating one processor's worth of cycles and
          starving everyone else. Actually I suppose "blocking" would be more
          accurate, since there is spare CPU capacity. When this happens, we
          typically see one CPU hovering around 70-80% and the other 20-30%,
          adding up to exactly 50% for the java process.

          This doesn't appear to be a garbage collection issue, as we've been
          running with verbose GC and pauses are under 5 seconds or so.
          • 2. Re: Weblogic hangs
            3004
            Thanks for the reply.
            First things first: You are (or someone is) spinning off a large number of
            threads. Any explanation on those?
            Yeah, that's us. We entirely removed that code, but it didn't help.
            However, it appears like you are locked up / blocked up on some reflection
            work that you are doing.

            Any more info?
            We fixed it, but I don't really understand it. We're spitting out
            some pretty large XML files and there was some code borrowed from
            Microsoft that was going through each byte and reformatting spaces or
            some such (changing "+" characters to "%20"). We didn't need this
            "feature", so removed it. Poof, the problem went away.

            Sure, this was crappy code that was looping through some huge strings,
            but I still don't understand why that'd bring a supposedly preemptive
            multitasking system to its knees. Even on an 8 CPU machine with 50
            execution threads this caused all other threads to basically be
            starved. Even the Weblogic monitor stopped receiving updates, ditto
            weblogic pings.

            It didn't appear to be a typical blocking issue, as the other threads
            were listed as runnable rather than waiting on monitor... but they
            weren't getting any CPU time, even though most CPUs were nearly idle.

            Anyway, thanks again for the reply. It's "fixed" and we've moved onto
            other things, but still somewhat of a mystery.
            • 3. RE: Weblogic hangs
              3004
              Were you able to resolve this problem of processor hanging at 50%?
              • 4. Re: Weblogic hangs
                3004
                "Raj Vibhakar" <rajbharat.vibhakar@gm.com> wrote in message news:40717e97$1@newsgroups.bea.com...
                Were you able to resolve this problem of processor hanging at 50%?
                Raj,

                You greatly reduce chances of getting an answer by not
                quoting the original question.

                Slava