4 Replies Latest reply on Jan 28, 2011 1:44 AM by user161771 - oracle

    Locked thread issue

    613820
      We encountered the following locked thread issue. Refer to the thread dump logs.
      Any idea on the root cause of the issue? It seens to us that it is the weblogic app server codes that are causing the locked threads.
      Will appreciate any advise.

      "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=22 idx=0x50 tid=329 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x9a73910[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1674)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1775)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1311)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1258)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2455)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

      -- end of trace

      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x58 tid=331 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x9a73910[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1674)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1775)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1311)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1258)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2455)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

      -- end of trace

      "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x54 tid=330 prio=5 alive, in native, daemon

      at jrockit/ext/epoll/EPoll.epollWait0(IIII)I(Native Method)

      at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:103)

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)

      ^-- Holding lock: java/lang/String@0x9a73910[fat lock]

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

      -- end of trace
        • 1. Re: Locked thread issue
          613820
          Any updates?
          • 2. Re: Locked thread issue
            user161771 - oracle
            This is standard behavior for the socket muxer threads. Only one of the epoll muxer threads is actually holding the lock at any one time. Here is the output of part of my thread dump on Linux with WLS 10.3.4 on JRockit on startup. The server is not hung up at all. Make a request on the server and take another thread dump and you'll see the lock change.

            1st dump with ExecuteThread 1 waiting to get the lock from ExecuteThread 2
            "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x58 tid=26836 prio=5 alive, blocked, native_blocked, daemon
                -- Blocked trying to get lock: java/lang/String@0xe82d6ec0[thin lock]
                at jrockit/vm/Threads.sleep(I)V(Native Method)
                at jrockit/vm/Locks.waitForThinRelease(Locks.java:955)
                at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1083)
                at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)
                at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
                at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
                at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
                at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
                at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
                at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
                at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                -- end of trace
            
            "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x5c tid=26837 prio=5 alive, in native, daemon
                at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
                at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
                at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
                ^-- Holding lock: java/lang/String@0xe82d6ec0[thin lock]
                at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
                at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
                at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
                at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
                at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                -- end of trace
            2nd dump after the server servers a request with ExecuteThread 2 waiting to get the lock from ExecuteThread 1
            "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x58 tid=26836 prio=5 alive, in native, daemon
                at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
                at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
                at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
                ^-- Holding lock: java/lang/String@0xe82d6ec0[fat lock]
                at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
                at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
                at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
                at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
                at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                -- end of trace
            
            "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x5c tid=26837 prio=5 alive, blocked, native_blocked, daemon
                -- Blocked trying to get lock: java/lang/String@0xe82d6ec0[fat lock]
                at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
                at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)
                at jrockit/vm/Locks.lockFat(Locks.java:1512)
                at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)
                at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)
                at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
                at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
                at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
                at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
                at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
                at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
                at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                -- end of trace
            Here is a blog on this topic: http://jojovedder.blogspot.com/2009/05/weblogic-socket-muxers-are-not-stuck.html
            • 3. Re: Locked thread issue
              829358
              James, I read your blog often - and now you've linked an oracle forum thread to mine :)

              - Jose (aka jojovedder)
              • 4. Re: Locked thread issue
                user161771 - oracle
                It was a nice write-up :) Thanks for doing that. We just are getting better about WebLogic Server blog posts and building out the community. Check out the @OracleWebLogic twitter handle. If you post new blog posts on an account we follow, we can retweet your posts.
                http://twitter.com/#!/OracleWebLogic