Forum Stats

  • 3,814,247 Users
  • 2,258,849 Discussions
  • 7,892,633 Comments

Discussions

OSB 12.2 - Cannot Activate session in service-bus console

Sanae
Sanae Member Posts: 49 Red Ribbon
edited Feb 5, 2019 12:10PM in SOA Suite Discusssions

Hello Guys,

 

When I try to activate the session in the service-bus console, the task take an infinity of time and generate a Stuck Thread and the state of AdminServer change to Warning

I applied the steps

1. I created the backup of the session folder ;

2. I deleted All Contents in the sessions folders;

3. I created the backup of each core_<osb-managed-server>

4. I created a new empty core_<osb-managed-server>

5. I deleted tmp cache stage from each server

6. Restart the AdminServer and Managed Server

When I retry to activate a new session in osb console, It take a infinity of time and in the logs ,  a STUCK thread is created :

<BEA-000337> <[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "719" seconds working on the request "Http Request Information: [email protected][POST /servicebus/faces/resources]

Any suggestions /solutions, please?

Best Regards,

Sanae

Answers

  • Raj__K
    Raj__K Member Posts: 412 Silver Badge
    edited Feb 5, 2019 7:03AM

    Hi, Delete the cache and tmp directories would be enough if you need to do clean up. Why you deleted other directories?

    Regards,

    G Raj

  • Sanae
    Sanae Member Posts: 49 Red Ribbon
    edited Feb 5, 2019 10:41AM

    Hello,

    The   delete of the tmp cache folders not resolve the issue

    That why , I tried to delete  the  session folder , but that not resolve the issue

    With the Support , they recommend to rename the edit.lock , but that not resolve the issue

    When I modify the url of any OSB  service and activate the session  from   service-bus  console  , the following logs appears in the AdminServer Logs :

    <AdminServer> <[STANDBY] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <13dc5cb6-b5ca-405f-b193-95fbfda65367-000002d4> <1548693491458> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320068> <Watch "StuckThread" in module "Module-FMWDFW" with severity "Notice" on server "AdminServer" has triggered at Jan 28, 2019 5:38:11 PM CET. Notification details:

    WatchRuleType: Log

    WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-000337') or (log.messageId == 'BEA-000337'))

    WatchData: MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "659" seconds working on the request "Http Request Information: [email protected][POST /servicebus/faces/resources]

    ", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

    java.lang.Object.wait(Native Method)

    weblogic.management.provider.internal.ActivateTaskImpl.waitForCompletion(ActivateTaskImpl.java:500)

    weblogic.management.provider.internal.ActivateTaskImpl.waitForTaskCompletion(ActivateTaskImpl.java:478)

    weblogic.management.provider.internal.EditAccessImpl.activateChangesAndWaitForCompletion(EditAccessImpl.java:1720)

    weblogic.management.mbeanservers.edit.internal.ConfigurationManagerMBeanImpl.activate(ConfigurationManagerMBeanImpl.java:435)

    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    java.lang.reflect.Method.invoke(Method.java:498)

    weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:532)

    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)

    com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)

    java.security.AccessController.doPrivileged(Native Method)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)

    weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:249)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)

    java.security.AccessController.doPrivileged(Native Method)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)

    weblogic.management.mbeanservers.edit.internal.RecordingInterceptor.invoke(RecordingInterceptor.java:178)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)

    java.security.AccessController.doPrivileged(Native Method)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)

    weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)

    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)

    java.security.AccessController.doPrivileged(Native Method)

    Any propositions, please?

    Best Regards ,

    Sanae BEKKAR

  • Raj__K
    Raj__K Member Posts: 412 Silver Badge
    edited Feb 5, 2019 11:46AM

    Hi,

    The Exception shown is for stuck thread, may not be actual Exception.

    With the Support , they recommend to rename the edit.lock , but that not resolve the issue 

    Are you referring Support from Oracle?

    I would not recommend deleting directories other than Temp and cache(After bringing down the server). You could check with Oracle support team for their recommendation.

    Instead of the OSB change, if you have any other changes (for ex disabling/enabling transaction property), is the change getting activated.

  • Sanae
    Sanae Member Posts: 49 Red Ribbon
    edited Feb 5, 2019 12:10PM

    Hi,

    Yes, I refer to Oracle Support

    As you propose, I trying just  click   to  Lock & Edit  button from the WebLogic console, and that take an infinity of time  and  the AdminServer take a Warning  state , I share you a part  of the Dump Theads :


      "#EM_SYSTEM_POOL#:StuckThreadDetection" #156 daemon prio=5 os_prio=0 tid=0x00000000083f1000 nid=0x1967 waiting on condition [0x00007f352594b000]
      
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      
      at java.lang.Thread.sleep(Native Method)
      
      at oracle.sysman.core.common.workmanager.Work.handleScheduledWork(Work.java:453)
      
      at oracle.sysman.core.common.workmanager.Work.call(Work.java:284)
      
      at oracle.sysman.core.common.workmanager.Work.call(Work.java:50)
      
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      
      at java.lang.Thread.run(Thread.java:748)
      
      "OAM REST Config Reload" #153 daemon prio=5 os_prio=0 tid=0x0000000007e11800 nid=0x1963 in Object.wait() [0x00007f3525a4c000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at oracle.security.idm.common.rest.IdmServletContainer$ServletReloader.run(IdmServletContainer.java:269)
      
      - locked <0x00000000bd044e48> (a oracle.security.idm.common.rest.IdmServletContainer$ServletReloader)
      
      "Configuration Store Observer" #154 daemon prio=5 os_prio=0 tid=0x0000000007dee000 nid=0x1962 in Object.wait() [0x00007f3528026000]
      
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.util.TimerThread.mainLoop(Timer.java:552)
      
      - locked <0x00000000bd045148> (a java.util.TaskQueue)
      
      at java.util.TimerThread.run(Timer.java:505)
      
      "pool-23-thread-1" #150 prio=5 os_prio=0 tid=0x0000000005f66000 nid=0x1958 waiting on condition [0x00007f352a127000]
      
      java.lang.Thread.State: WAITING (parking)
      
      at sun.misc.Unsafe.park(Native Method)
      
      - parking to wait for <0x00000000b8bb1ab8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      
      at java.lang.Thread.run(Thread.java:748)
      
      "Thread-60" #149 daemon prio=5 os_prio=0 tid=0x0000000006fc0000 nid=0x1955 waiting on condition [0x00007f352aa28000]
      
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      
      at java.lang.Thread.sleep(Native Method)
      
      at com.bea.alsb.alert.log.purger.ReliableCommandExecutor$PollerThread.run(ReliableCommandExecutor.java:32)
      
      "OAM REST Config Reload" #146 daemon prio=5 os_prio=0 tid=0x0000000004ad3800 nid=0x1953 in Object.wait() [0x00007f352ab29000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at oracle.security.idm.common.rest.IdmServletContainer$ServletReloader.run(IdmServletContainer.java:269)
      
      - locked <0x00000000b685ed88> (a oracle.security.idm.common.rest.IdmServletContainer$ServletReloader)
      
      "Configuration Store Observer" #147 daemon prio=5 os_prio=0 tid=0x000000000645f800 nid=0x1952 in Object.wait() [0x00007f352cd30000]
      
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.util.TimerThread.mainLoop(Timer.java:552)
      
      - locked <0x00000000b685f088> (a java.util.TaskQueue)
      
      at java.util.TimerThread.run(Timer.java:505)
      
      "WsMgmtWorkScheduler" #145 daemon prio=5 os_prio=0 tid=0x0000000006590000 nid=0x1951 waiting on condition [0x00007f3554293000]
      
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      
      at java.lang.Thread.sleep(Native Method)
      
      at oracle.j2ee.ws.common.mgmt.util.WorkSchedulerImpl$1.run(WorkSchedulerImpl.java:37)
      
      "[email protected] 12.2.1.3.0" #142 daemon prio=3 os_prio=0 tid=0x00000000048a3000 nid=0x194c in Object.wait() [0x00007f352cc2f000]
      
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:460)
      
      at com.oracle.common.base.Blocking.wait(Blocking.java:97)
      
      at com.oracle.common.base.Blocking.wait(Blocking.java:80)
      
      at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:9)
      
      - locked <0x00000000b591cb68> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
      
      at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
      
      at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
      
      at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:43)
      
      at java.lang.Thread.run(Thread.java:748)
      
      "[STANDBY] ExecuteThread: '54' for queue: 'weblogic.kernel.Default (self-tuning)'" #138 daemon prio=5 os_prio=0 tid=0x0000000005bdf800 nid=0x1948 in Object.wait() [0x00007f352d431000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b2ecd4f8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'" #137 daemon prio=5 os_prio=0 tid=0x0000000005baf000 nid=0x1947 in Object.wait() [0x00007f352d532000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b2ecd6e8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "com.bea.osb.transports.jms.TimerThread" #136 daemon prio=5 os_prio=0 tid=0x00000000033ef800 nid=0x1942 waiting on condition [0x00007f3530458000]
      
      java.lang.Thread.State: WAITING (parking)
      
      at sun.misc.Unsafe.park(Native Method)
      
      - parking to wait for <0x00000000b1ab9e30> (a java.util.concurrent.Semaphore$NonfairSync)
      
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      
      at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
      
      at com.bea.wli.sb.transports.jms.JmsAsyncMessageTable$TimeoutThread.run(JmsAsyncMessageTable.java:258)
      
      "Thread-57" #135 daemon prio=5 os_prio=0 tid=0x0000000002e97000 nid=0x193f in Object.wait() [0x00007f3531559000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
      
      - locked <0x00000000b101e680> (a java.lang.ref.ReferenceQueue$Lock)
      
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
      
      at com.bea.wli.config.derivedcache.impl.GarbageCollectionNotifier.run(GarbageCollectionNotifier.java:165)
      
      at java.lang.Thread.run(Thread.java:748)
      
      "[ACTIVE] ExecuteThread: '52' for queue: 'weblogic.kernel.Default (self-tuning)'" #134 daemon prio=5 os_prio=0 tid=0x0000000002b04000 nid=0x1930 waiting on condition [0x00007f3532117000]
      
      java.lang.Thread.State: WAITING (parking)
      
      at sun.misc.Unsafe.park(Native Method)
      
      - parking to wait for <0x0000000085832da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      
      at weblogic.rjvm.ResponseImpl.waitForDataInternal(ResponseImpl.java:138)
      
      at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:120)
      
      at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:190)
      
      at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:129)
      
      at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:299)
      
      at javax.management.remote.rmi.RMIConnectionImpl_12213_WLStub.fetchNotifications(Unknown Source)
      
      at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1689)
      
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
      
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
      
      at weblogic.security.Security.runAs(Security.java:62)
      
      at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1687)
      
      at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
      
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      at java.lang.reflect.Method.invoke(Method.java:498)
      
      at weblogic.management.remote.common.InvocationContextProxyHandler.invoke(ClientProviderBase.java:675)
      
      at com.sun.proxy.$Proxy259.fetchNotifications(Unknown Source)
      
      at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1341)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:596)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
      
      at weblogic.management.mbeanservers.domainruntime.internal.JMXExecutor$1.timerExpired(JMXExecutor.java:97)
      
      at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:301)
      
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
      
      at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
      
      at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
      
      at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
      
      at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
      
      at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
      
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
      
      "[STANDBY] ExecuteThread: '51' for queue: 'weblogic.kernel.Default (self-tuning)'" #133 daemon prio=5 os_prio=0 tid=0x0000000002b02800 nid=0x192f in Object.wait() [0x00007f3532218000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09be238> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '50' for queue: 'weblogic.kernel.Default (self-tuning)'" #132 daemon prio=5 os_prio=0 tid=0x0000000002b00800 nid=0x192e in Object.wait() [0x00007f3532319000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09be428> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '49' for queue: 'weblogic.kernel.Default (self-tuning)'" #131 daemon prio=5 os_prio=0 tid=0x0000000002aff000 nid=0x192d in Object.wait() [0x00007f353241a000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09be618> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '48' for queue: 'weblogic.kernel.Default (self-tuning)'" #130 daemon prio=5 os_prio=0 tid=0x0000000002afd000 nid=0x192c in Object.wait() [0x00007f353251b000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09be808> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'" #129 daemon prio=5 os_prio=0 tid=0x0000000002ae7800 nid=0x192b in Object.wait() [0x00007f3532681000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09be9f8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[ACTIVE] ExecuteThread: '46' for queue: 'weblogic.kernel.Default (self-tuning)'" #128 daemon prio=5 os_prio=0 tid=0x0000000002aab000 nid=0x192a in Object.wait() [0x00007f3532782000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bebe8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '45' for queue: 'weblogic.kernel.Default (self-tuning)'" #127 daemon prio=5 os_prio=0 tid=0x0000000002aa9800 nid=0x1929 in Object.wait() [0x00007f3532883000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bedd8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'" #126 daemon prio=5 os_prio=0 tid=0x0000000002a4f800 nid=0x1928 in Object.wait() [0x00007f3532984000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09befc8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '43' for queue: 'weblogic.kernel.Default (self-tuning)'" #125 daemon prio=5 os_prio=0 tid=0x0000000002a4e800 nid=0x1927 in Object.wait() [0x00007f3532a85000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bf1b8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[ACTIVE] ExecuteThread: '42' for queue: 'weblogic.kernel.Default (self-tuning)'" #124 daemon prio=5 os_prio=0 tid=0x00000000029d6800 nid=0x1925 waiting on condition [0x00007f3532c37000]
      
      java.lang.Thread.State: WAITING (parking)
      
      at sun.misc.Unsafe.park(Native Method)
      
      - parking to wait for <0x0000000084226608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      
      at weblogic.rjvm.ResponseImpl.waitForDataInternal(ResponseImpl.java:138)
      
      at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:120)
      
      at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:190)
      
      at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:129)
      
      at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:299)
      
      at javax.management.remote.rmi.RMIConnectionImpl_12213_WLStub.fetchNotifications(Unknown Source)
      
      at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1689)
      
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
      
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
      
      at weblogic.security.Security.runAs(Security.java:62)
      
      at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1687)
      
      at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
      
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      at java.lang.reflect.Method.invoke(Method.java:498)
      
      at weblogic.management.remote.common.InvocationContextProxyHandler.invoke(ClientProviderBase.java:675)
      
      at com.sun.proxy.$Proxy259.fetchNotifications(Unknown Source)
      
      at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1341)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:596)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
      
      at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
      
      at weblogic.management.mbeanservers.domainruntime.internal.JMXExecutor$1.timerExpired(JMXExecutor.java:97)
      
      at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:301)
      
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
      
      at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
      
      at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
      
      at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
      
      at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
      
      at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
      
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
      
      "[STANDBY] ExecuteThread: '41' for queue: 'weblogic.kernel.Default (self-tuning)'" #123 daemon prio=5 os_prio=0 tid=0x0000000002969000 nid=0x1924 in Object.wait() [0x00007f3532d38000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bf5c8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'" #122 daemon prio=5 os_prio=0 tid=0x0000000002967000 nid=0x1923 in Object.wait() [0x00007f3532e39000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bf7b8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" #121 daemon prio=5 os_prio=0 tid=0x0000000002947800 nid=0x1922 in Object.wait() [0x00007f3532f7f000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bf9a8> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" #120 daemon prio=5 os_prio=0 tid=0x0000000002945800 nid=0x1921 in Object.wait() [0x00007f353309d000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bfb98> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #119 daemon prio=5 os_prio=0 tid=0x00000000028fd000 nid=0x1920 in Object.wait() [0x00007f35331dc000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bfd88> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" #118 daemon prio=5 os_prio=0 tid=0x00000000028a4000 nid=0x191f in Object.wait() [0x00007f3533366000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09bff78> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" #117 daemon prio=5 os_prio=0 tid=0x00000000025e4800 nid=0x191e in Object.wait() [0x00007f3533524000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09c0168> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" #116 daemon prio=5 os_prio=0 tid=0x000000000266a800 nid=0x191d in Object.wait() [0x00007f3533671000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09c0358> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" #115 daemon prio=5 os_prio=0 tid=0x0000000001ffa800 nid=0x191c in Object.wait() [0x00007f353378e000]
      
      java.lang.Thread.State: WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at java.lang.Object.wait(Object.java:502)
      
      at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:323)
      
      - locked <0x00000000b09c0548> (a weblogic.work.ExecuteThread)
      
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:383)
      
      "[STUCK] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" #114 daemon prio=1 os_prio=0 tid=0x0000000001e0f800 nid=0x191b in Object.wait() [0x00007f35338e6000]
      
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      
      at java.lang.Object.wait(Native Method)
      
      at weblogic.management.provider.internal.ActivateTaskImpl.waitForCompletion(ActivateTaskImpl.java:500)
      
      - locked <0x00000000c63da660> (a weblogic.management.provider.internal.ActivateTaskImpl)
      
      at weblogic.management.provider.internal.ActivateTaskImpl.waitForTaskCompletion(ActivateTaskImpl.java:478)
      
      at weblogic.management.provider.internal.EditAccessImpl.activateChangesAndWaitForCompletion(EditAccessImpl.java:1720)
      
      at weblogic.management.mbeanservers.edit.internal.ConfigurationManagerMBeanImpl.activate(ConfigurationManagerMBeanImpl.java:435)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      at java.lang.reflect.Method.invoke(Method.java:498)
      
      at weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:532)
      
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
      
      at java.security.AccessController.doPrivileged(Native Method)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
      
      at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:249)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
      
      at java.security.AccessController.doPrivileged(Native Method)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
      
      at weblogic.management.mbeanservers.edit.internal.RecordingInterceptor.invoke(RecordingInterceptor.java:178)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
      
      at java.security.AccessController.doPrivileged(Native Method)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
      
      at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
      
      at java.security.AccessController.doPrivileged(Native Method)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
      
      at weblogic.management.mbeanservers.edit.internal.EditLockInterceptor.invoke(EditLockInterceptor.java:135)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$21.run(WLSMBeanServerInterceptorBase.java:589)
      
      at java.security.AccessController.doPrivileged(Native Method)
      
      at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:587)
      

    Any propositions, please?

    Thanks

    Best Regards

    Sanae

  • AmitRey
    AmitRey Member Posts: 25 Blue Ribbon

    We have the same problem. Reemoved temp, cache. Restarted admin, managed servers. But it didn't resolve the issue. Whenver trying to make a change, it shows "in progress" for a long time and prints below error message into log. Any guidance how to debug?

     <BEA-000337> <[STUCK] ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "662" seconds working on the request "Http Request Information: [email protected][POST /servicebus/faces/resources]

    ", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

        java.lang.Object.wait(Native Method)

        weblogic.management.provider.internal.ActivateTaskImpl.waitForCompletion(ActivateTaskImpl.java:500)

        weblogic.management.provider.internal.ActivateTaskImpl.waitForTaskCompletion(ActivateTaskImpl.java:478)