3 Replies Latest reply: Jul 18, 2014 3:18 PM by Luz Mestre-Oracle RSS

    weblogic 11.1.1.4.0 FULL THREAD DUMP issue Oracle JRockit(R) R28.1

    915538

      Hi,

       

      We are using WebLogic 11.1.1.4.0 version. We are facing of full thread dump in weblogic server. Due to this thread dump issue the bpel processes are getting stuck in the middle.

       

      Can anyone please guide with resolution to overcome this issue.

       

      Error:

      ####<Jun 27, 2014 4:58:28 PM PDT> <Notice> <WebLogicServer> <kpch3.cbf.corp.google.com> <soa_server2> <Thread-380> <weblogic> <> <> <1403913508622> <BEA-000378> <Server failed to shutdown within the configured timeout of 120 seconds. The server process will exit now.>####<Jun 27, 2014 4:58:28 PM PDT> <Error> <> <kpch3.cbf.corp.google.com> <soa_server2> <Thread-380> <weblogic> <> <> <1403913508751> <BEA-000000> <===== FULL THREAD DUMP ===============Fri Jun 27 16:58:28 2014Oracle JRockit(R) R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64"Main Thread" id=1 idx=0x4 tid=7558 prio=5 alive, waiting, native_blocked   -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1047e29f8[fat lock]   at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)   at java/lang/Object.wait(J)V(Native Method)   at java/lang/Object.wait(Object.java:485)   at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:979)   ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1047e29f8[fat lock]   at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:488)   at weblogic/Server.main(Server.java:71)   at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)   -- end of traceOpen lock chains================Chain 1:"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0xb4 tid=7750 waiting for java/lang/String@0x106de26e0 held by:"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=25 idx=0xb0 tid=7749 (active)===== END OF THREAD DUMP ===============


      -Thanks

      GVS

        • 1. Re: weblogic 11.1.1.4.0 FULL THREAD DUMP issue Oracle JRockit(R) R28.1
          Luz Mestre-Oracle

          1. It sounds like the weblogic server is attempted to be stop and the a thread dump is generated (thread dump  during shut down is only collected when the server is not able to be  gracefully shut down), What is happening before this? Was the server manually stopped?

          2. What JDK is used? (java -version)

          Best Regards

          Luz

          • 2. Re: weblogic 11.1.1.4.0 FULL THREAD DUMP issue Oracle JRockit(R) R28.1
            915538

            Hi Luz,

             

            I would like to thank for the response.

             

            Server is restarted on 06/25. But this issue caused on 06/27.

             

            Below is the java version we are using :

            java version "1.6.0_19"

            Java(TM) SE Runtime Environment (build 1.6.0_19-b04)

            Java HotSpot(TM) Server VM (build 16.2-b04, mixed mode)

             

            LOG details :

            ####<Jun 27, 2014 4:58:07 PM PDT> <Info> <Common> <kpch3.cbf.corp.google.com> <soa_server2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913487422> <BEA-000628> <Created "1" resources for pool "SOADataSource-rac1", out of which "1" are available and "0" are unavailable.>

            ####<Jun 27, 2014 4:58:25 PM PDT> <Info> <JDBC> <kpch3.cbf.corp.google.com> <soa_server2> <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913505573> <BEA-001128> <Connection for pool "SOADataSource-rac0" closed.>

            ####<Jun 27, 2014 4:58:28 PM PDT> <Notice> <WebLogicServer> <kpch3.cbf.corp.google.com> <soa_server2> <Thread-380> <weblogic> <> <> <1403913508622> <BEA-000378> <Server failed to shutdown within the configured timeout of 120 seconds. The server process will exit now.>

            ####<Jun 27, 2014 4:58:28 PM PDT> <Error> <> <kpch3.cbf.corp.google.com> <soa_server2> <Thread-380> <weblogic> <> <> <1403913508751> <BEA-000000> <

            ===== FULL THREAD DUMP ===============

            Fri Jun 27 16:58:28 2014

            Oracle JRockit(R) R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64

             

            "Main Thread" id=1 idx=0x4 tid=7558 prio=5 alive, waiting, native_blocked

                -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1047e29f8[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)

                at java/lang/Object.wait(Object.java:485)

                at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:979)

                ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1047e29f8[fat lock]

                at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:488)

                at weblogic/Server.main(Server.java:71)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

             

            "(Signal Handler)" id=2 idx=0x8 tid=7559 prio=5 alive, native_blocked, daemon

             

            "(OC Main Thread)" id=3 idx=0xc tid=7560 prio=5 alive, native_waiting, daemon

             

            "(GC Worker Thread 1)" id=? idx=0x10 tid=7561 prio=5 alive, daemon

             

            "(GC Worker Thread 2)" id=? idx=0x14 tid=7562 prio=5 alive, daemon

             

            "(GC Worker Thread 3)" id=? idx=0x18 tid=7563 prio=5 alive, daemon

             

            "(GC Worker Thread 4)" id=? idx=0x1c tid=7564 prio=5 alive, daemon

             

            "(GC Worker Thread 5)" id=? idx=0x20 tid=7565 prio=5 alive, daemon

             

            ......

            .......

            "weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@49e6697b" id=433 idx=0x588 tid=15843 prio=10 alive, in native, daemon

                at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)

                at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]

                at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]

                at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]

                at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:718)[optimized]

                at oracle/net/ns/Packet.receive(Packet.java:295)[optimized]

                at oracle/net/ns/DataPacket.receive(DataPacket.java:106)[optimized]

                at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:317)[optimized]

                at oracle/net/ns/NetInputStream.read(NetInputStream.java:262)[optimized]

                at oracle/net/ns/NetInputStream.read(NetInputStream.java:187)[inlined]

                at oracle/net/ns/NetInputStream.read(NetInputStream.java:104)[optimized]

                at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)[inlined]

                at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)[optimized]

                at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)[optimized]

                at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)[inlined]

                at oracle/jdbc/driver/T4CTTIfun.receive(T4CTTIfun.java:312)[optimized]

                at oracle/jdbc/driver/T4CTTIfun.doRPC(T4CTTIfun.java:204)[optimized]

                at oracle/jdbc/driver/T4Caqdq.doOAQDQ(T4Caqdq.java:183)[optimized]

                at oracle/jdbc/driver/T4CConnection.doDequeue(T4CConnection.java:4559)[optimized]

                ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x13b514788[thin lock]

                at oracle/jdbc/driver/PhysicalConnection.dequeue(PhysicalConnection.java:17726)[optimized]

                at oracle/jdbc/driver/PhysicalConnection.dequeue(PhysicalConnection.java:17812)[optimized]

                at oracle/jdbc/OracleConnectionWrapper.dequeue(OracleConnectionWrapper.java:1606)[optimized]

                at weblogic/jdbc/wrapper/XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Ljava/lang/String;Loracle/jdbc/aq/AQDequeueOptions;Ljava/lang/String;)Loracle/jdbc/aq/AQMessage;(Unknown Source)[optimized]

                at weblogic/jdbc/wrapper/JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Ljava/lang/String;Loracle/jdbc/aq/AQDequeueOptions;Ljava/lang/String;)Loracle/jdbc/aq/AQMessage;(Unknown Source)[optimized]

                at oracle/tip/adapter/aq/v2/database/AbstractDequeueAgent.dequeue(AbstractDequeueAgent.java:375)[optimized]

                at oracle/tip/adapter/aq/v2/database/AbstractDequeueAgent.dequeueMessage(AbstractDequeueAgent.java:277)[inlined]

                at oracle/tip/adapter/aq/v2/database/AbstractDequeueAgent.getInputRecord(AbstractDequeueAgent.java:250)[optimized]

                at oracle/tip/adapter/aq/v2/database/AbstractDequeueAgent.run(AbstractDequeueAgent.java:111)

                at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:120)

                at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)

                at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

             

            "orabpel.invoke.pool-4.thread-56" id=759 idx=0x590 tid=26436 prio=5 alive, parked, native_blocked

                -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x11e824050

                at jrockit/vm/Locks.park0(J)V(Native Method)

                at jrockit/vm/Locks.park(Locks.java:2230)[inlined]

                at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]

                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[optimized]

                at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]

                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java/lang/Thread.run(Thread.java:619)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

             

             

             

            ####<Jun 27, 2014 4:59:15 PM PDT> <Info> <WebLogicServer> <kpch3.cbf.corp.google.com> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1403913555018> <BEA-000000> <WebLogic Server "soa_server2" version:

            WebLogic Server 10.3.4.0  Fri Dec 17 20:47:33 PST 2010 1384255  Copyright (c) 1995, 2009, Oracle and/or its affiliates. All rights reserved.>

            ####<Jun 27, 2014 4:59:15 PM PDT> <Notice> <Log Management> <kpch3.cbf.corp.google.com> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1403913555163> <BEA-170019> <The server log file /u1/app/oracle/admin/soa_domain/managedservers/soa_domain/servers/soa_server2/logs/soa_server2.log is opened. All server side log events will be written to this file.>

             

            -Thanks

            GVS

            • 3. Re: weblogic 11.1.1.4.0 FULL THREAD DUMP issue Oracle JRockit(R) R28.1
              Luz Mestre-Oracle

              The thread dump is collected to show that this thread was still running when the server was shut down.

              So, could you confirm what is occurring before this?

              Are you attempting to restart the server? or

              Was the server not responding, then the server was attempted to be shut down?

               

              In any case, when weblogic server is not responding, you can collect several thread dumps 10-20 secs apart from each other and analyze them with a tool like thread logic.

              https://java.net/projects/threadlogic

              That can give you an idea of what subsystem is not responding.

              I hope this helps.

              Best Regards

              Luz