This discussion is archived
13 Replies Latest reply: Dec 16, 2013 6:15 AM by 869615 RSS

WebLogic intermittent issues while shutting down, migrating JMS or stoping/starting application

WLS_usr Newbie
Currently Being Moderated

Hi WebLogic Gurus,

                            For a month now, we have been running into issue (intermittent) when stopping WLS manage server instance (mostly Manage server 1) or migrating JMS server from Manage server 2 to Manage server 1 or starting up application on Cluster (stack trace is different for all of the above task). All of the above happens suddenly and only way to mitigate the issue is to kill manage server 1. Once killed and started back, it works ok for a while i.e we can stop the manage server instance, migrate jms server instance and start/stop application but it comes back again after few days (1 or two mostly).

Here is the details of our enironment..

 

WLS: 10.3.4 (cluster with MS1, MS2, admin all located on different hosts)

OS: HP UX 11.31

JMS service: One JMS server targeted to migratable target

Queues/topics: secured using policies and distributed in nature

Global trust: enabled as other WLS domain application makes jms calls.

 

Here is the error we get while shutting down manage server instance:

 

####<Jul 18, 2013 11:55:19 AM MDT> <Info> <Server> <Host1> <Manage_server_1> <[ACTIVE] ExecuteThread: '2' f

or queue: 'weblogic.kernel.Default (self-tuning)'> <WLadminuser> <> <> <1374170119436> <BEA-002639> <ServerRuntime:java.la

ng.SecurityException: User: 'WLadminuser' does not have permission to shutdown server

  at weblogic.t3.srvr.T3Srvr.checkPrivileges(T3Srvr.java:1278)

  at weblogic.t3.srvr.T3Srvr.checkShutdownPrivileges(T3Srvr.java:1254)

  at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:1020)

  at weblogic.t3.srvr.ServerRuntime.forceShutdown(ServerRuntime.java:351)

  at weblogic.server.RemoteLifeCycleOperationsImpl.forceShutdown(RemoteLifeCycleOperationsImpl.java:57)

  at weblogic.server.RemoteLifeCycleOperationsImpl_WLSkel.invoke(Unknown Source)

  at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)

  at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)

  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)

  at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)

  at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)

  at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)

  at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)

  at weblogic.work.ExecuteThread.run(ExecuteThread.java:176

 

When enabled ATN,ATZ flag, it sees "WLadminuser" user but receives "indeterminate" message and therefore access to shutdown is denied as not having enough privillages.

 

####<Jul 18, 2013 11:55:19 AM MDT> <Debug> <SecurityAtz> <Host1> <Manage_server_1> <[ACTIVE] ExecuteThread:

'2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1374170119435> <BEA-000000> <XACML Author

ization isAccessAllowed() indeterminate: com.bea.common.security.xacml.DocumentParseException: org.xml.sax.SAXParseExcep

tion: The entity "lt" was referenced, but not declared.>

 

####<Jul 18, 2013 11:55:19 AM MDT> <Debug> <SecurityAtz> <Host1> <Manage_server_1> <[ACTIVE] ExecuteThread:

'2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1374170119435> <BEA-000000> <XACML Author

ization isAccessAllowed() indeterminate: com.bea.common.security.xacml.DocumentParseException: org.xml.sax.SAXParseExcep

tion: The entity "lt" was referenced, but not declared.>

 

This below stack trace is while migrating jms service from manage server 2 to 1:

 

####<Jul 15, 2013 9:33:58 AM MDT> <Info> <Management> <admin_host> <admin> <[ACTIVE] ExecuteThread:

'0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902438294> <BEA-141073> <Migration Tas

k (Migration of JMS_Mig_Target to Manage_server_1) - weblogic.cluster.migration.MigrationException: Exception - java.rmi.

RemoteException: Error trying to activate service JMS_Mig_Target: java.lang.AssertionError: ERROR: A managed object did

not have a setLogicalName method; nested exception is:

java.lang.AssertionError: ERROR: A managed object did not have a setLogicalName method>

####<Jul 15, 2013 9:33:58 AM MDT> <Info> <Management> <admin_host> <admin> <[ACTIVE] ExecuteThread:

'0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902438296> <BEA-141073> <Migration Tas

k (Migration of JMS_Mig_Target to Manage_server_1) - weblogic.management.runtime.MigrationException: weblogic.cluster.mig

ration.MigrationException: Exception - java.rmi.RemoteException: Error trying to activate service JMS_Mig_Target: java.l

ang.AssertionError: ERROR: A managed object did not have a setLogicalName method; nested exception is:

java.lang.AssertionError: ERROR: A managed object did not have a setLogicalName method>

####<Jul 15, 2013 9:33:58 AM MDT> <Info> <Management> <admin_host> <admin> <[ACTIVE] ExecuteThread:

'0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902438297> <BEA-141073> <Migration Tas

k (Migration of JMS_Mig_Target to Manage_server_1) - weblogic.management.runtime.MigrationException: weblogic.cluster.mig

ration.MigrationException: Exception - java.rmi.RemoteException: Error trying to activate service JMS_Mig_Target: java.l

ang.AssertionError: ERROR: A managed object did not have a setLogicalName method; nested exception is:

java.lang.AssertionError: ERROR: A managed object did not have a setLogicalName method

at weblogic.cluster.migration.management.MigrationTask.failWith(MigrationTask.java:702)

at weblogic.cluster.migration.management.MigrationTask.migrate(MigrationTask.java:525)

at weblogic.cluster.migration.management.MigrationTask.access$300(MigrationTask.java:45)

at weblogic.cluster.migration.management.MigrationTask$1$1.run(MigrationTask.java:329)

at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

at weblogic.cluster.migration.management.MigrationTask$1.run(MigrationTask.java:320)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

 

 

This is while shutting down the application:

 

####<Jul 15, 2013 9:33:12 AM MDT> <Debug> <SecurityAtz> <admin_host> <admin> <[ACTIVE] ExecuteThread

: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902392572> <BEA-000000> <com.bea.com

mon.security.internal.service.AuthorizationServiceImpl.isAccessAllowed returning adjudicated: true>

####<Jul 15, 2013 9:33:12 AM MDT> <Info> <J2EE Deployment SPI> <admin_host> <admin> <[ACTIVE] Execut

eThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <WLSuser> <> <> <1373902392730> <BEA-260121> <Initiat

ing stop operation for application, app.Service#1.0 [archive: null], to manage_Cluster .>

####<Jul 15, 2013 9:33:16 AM MDT> <Info> <Deployer> <admin_host> <admin> <[ACTIVE] ExecuteThread: '4

' for queue: 'weblogic.kernel.Default (self-tuning)'> <WLSuser> <> <> <1373902396685> <BEA-149038> <Initiating Task fo

r Jms.Service [Version=1.0] : [Deployer:149026]stop application app.Service [Version=1.0] on manage_Cluster.>

####<Jul 15, 2013 9:33:16 AM MDT> <Warning> <Deployer> <admin_host> <admin> <[STANDBY] ExecuteThread

: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902396813> <BEA-149004> <Failures we

re detected while initiating stop task for application 'app.Service [Version=1.0]'.>

####<Jul 15, 2013 9:33:16 AM MDT> <Warning> <Deployer> <admin_host> <admin> <[STANDBY] ExecuteThread

: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1373902396814> <BEA-149078> <Stack trace

for message 149004

java.lang.Exception: java.lang.Error: java.lang.reflect.InvocationTargetException

at weblogic.deploy.service.internal.adminserver.AdminRequestStatus.includePrepareFailure(AdminRequestStatus.java

:1421)

at weblogic.deploy.service.internal.adminserver.AdminRequestStatus.signalCancelFailed(AdminRequestStatus.java:12

69)

at weblogic.deploy.service.internal.statemachines.adminserver.AwaitingCancelResponses.signalCancelCompletion(Awa

itingCancelResponses.java:126)

at weblogic.deploy.service.internal.statemachines.adminserver.AwaitingCancelResponses.doCancelCompletionCheck(Aw

aitingCancelResponses.java:112)

at weblogic.deploy.service.internal.statemachines.adminserver.AwaitingCancelResponses.receivedCancelFailed(Await

ingCancelResponses.java:97)

at weblogic.deploy.service.internal.adminserver.AdminRequestImpl.receivedCancelFailed(AdminRequestImpl.java:503)

at weblogic.deploy.service.internal.transport.CommonMessageReceiver.receiveCancelFailedMsg(CommonMessageReceiver

.java:321)

at weblogic.deploy.service.internal.transport.CommonMessageReceiver$3.run(CommonMessageReceiver.java:836)

at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

 

Regards.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points