This discussion is archived
1 Reply Latest reply: Mar 10, 2013 10:31 AM by souvik_das RSS

Nodemanager restarts a server that was shutdown normally

994279 Newbie
Currently Being Moderated
Nodemanager restarts a server that was shutdown normally.

We have java based nodemanager configured and we use console to start and stop weblogic server instances. Please find below nodemanager.properties file

#Wed Dec 12 14:09:24 EST 2012
NodeManagerHome=/opt/app/nodemanager/extwlsdva
ListenAddress=extwlsdva
ListenPort=5556
ListenBacklog=50
SecureListener=false
AuthenticationEnabled=true
NativeVersionEnabled=true
CrashRecoveryEnabled=false
JavaHome=/opt/app/jdk/jdk1.7.0_15/jre
StartScriptEnabled=true
StartScriptName=startWebLogic.sh
StopScriptEnabled=false
LogFile=/opt/app/nodemanager/extwlsdva/nodemanager.log
LogLevel=FINEST
LogLimit=0
LogCount=1
LogAppend=true
LogToStderr=true
LogFormatter=weblogic.nodemanager.server.LogFormatter
DomainsFile=/opt/app/nodemanager/extwlsdva/nodemanager.domains
DomainsFileEnabled=true
StateCheckInterval=500
UseMACBroadcast=false
DomainRegistrationEnabled=false
DomainsDirRemoteSharingEnabled=false
PropertiesVersion=10.3



We also have start/stop scripts that use wlcontrol.sh (script based nodemanager), and we also use these start/stop scripts to recycle the weblogic server instances. Please find below a sample startup command.

$WL_HOME/common/bin/wlscontrol.sh -d $DOMAIN_NAME -n /opt/app/nodemanager/$HOST -c -f $DOMAIN_HOME/bin/startWebLogic.sh -s $SERVER_NAME -r $DOMAIN_HOME START


We are observing below issue and wanted to know what the fix is.

1.     Weblogic.Nodemanager is started and is up and running successfully.
2.     Managed server 1a (ms1a) is started by the startup script (wlscontrol.sh). Managed Servers 2a (ms2a) is started using console.
3.     Nodemanager is recycled (weblogic.Nodemanager). It restarts fine and starts monitoring the active managed servers ms1a and ms2a
4.     Ms1a is brought down (either using script or console). Nodemanager thinks that Ms1a failed and restarts Ms1a.
5.     Ms2a is brought down (either using script or console). Nodemanager recognizes correctly that it was shutdown normally and does not restart Ms2a.

From Nodemanager Log: nodemanager.log
<Feb 25, 2013 11:01:20 AM> <INFO> <HostedDvlpDomain> <MgrSvr1a> <Server failed so attempting to restart (restart count = 1)>
<Feb 25, 2013 11:01:20 AM> <INFO> <HostedDvlpDomain> <MgrSvr1a> <Server failed so attempting to restart (restart count = 1)>
Feb 25, 2013 11:01:20 AM weblogic.nodemanager.server.AbstractServerManager log
INFO: Server failed so attempting to restart (restart count = 1)

From weblogic stdout log: MgrSvr1a.out
<Mon Feb 25 11:01:20 EST 2013)> <Info> <NodeManager> <Server start script exited>
<Mon Feb 25 11:01:20 EST 2013)> <Info> <NodeManager> <Server was shut down normally>
<Feb 25, 2013 11:01:20 AM> <FINEST> <NodeManager> <Waiting for the process to die: 12726>
<Feb 25, 2013 11:01:20 AM> <INFO> <NodeManager> <Server failed so attempting to restart (restart count = 1)>


Can you please review and let us know why in step4 above, nodemanager is restarting a weblogic server instance thinking that it failed when in fact it was shutdown normally.

Edited by: 991276 on Mar 1, 2013 10:34 AM
  • 1. Re: Nodemanager restarts a server that was shutdown normally
    souvik_das Newbie
    Currently Being Moderated
    NodeManagers are specific to a hardware instance/machine which can have more than one Managed Server(MS) running on the same.
    NodeManagers of every hardware instance/machine need to be enrolled with the Admin Server using the nmEnroll() WLST command.
    If a MS instance is under NM control then even start and stops via console are done using the NM.
    In your case first check using ptree command if the server life cycle is controlled by the NM process.
    If not then the NM is not properly associated with the instance and the admin server and that needs to be corrected.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points