Weblogic 10.3.5 Server start via wlst intermittent

2610779

Hi,

I am using a wlst script to restart the server instances once the libraries are deployed. The problem is that sometimes the server start works fine and at certain times it fails with the below error in the stacktrace.

Starting server mserver ....Server 'mserver' has already been started

No stack trace available.

This Exception occurred at Sun Apr 03 20:33:53 IST 2016.

weblogic.management.scripting.ScriptException: Error occured while performing start : Server with name mserver failed to be started

        at weblogic.management.scripting.ExceptionHandler.handleException(ExceptionHandler.java:48)

        at weblogic.management.scripting.LifeCycleHandler.startServer(LifeCycleHandler.java:502)

        at weblogic.management.scripting.WLScriptContext.start(WLScriptContext.java:523)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.python.core.PyReflectedFunction.__call__(Unknown Source)

        at org.python.core.PyMethod.__call__(Unknown Source)

        at org.python.core.PyObject.__call__(Unknown Source)

        at org.python.core.PyObject.invoke(Unknown Source)

        at org.python.pycode._pyx186.start$80(<iostream>:1272)

        at org.python.pycode._pyx186.call_function(<iostream>)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyFunction.__call__(Unknown Source)

        at org.python.pycode._pyx185.startWLServer$35(/home/wcpcluster/nithin/working/scripts/incDeploy.py:787)

        at org.python.pycode._pyx185.call_function(/home/wcpcluster/nithin/working/scripts/incDeploy.py)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyFunction.__call__(Unknown Source)

        at org.python.pycode._pyx185.f$0(/home/wcpcluster/nithin/working/scripts/incDeploy.py:866)

        at org.python.pycode._pyx185.call_function(/home/wcpcluster/nithin/working/scripts/incDeploy.py)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyCode.call(Unknown Source)

        at org.python.core.Py.runCode(Unknown Source)

        at org.python.util.PythonInterpreter.execfile(Unknown Source)

        at weblogic.management.scripting.WLST.main(WLST.java:124)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at weblogic.WLST.main(WLST.java:29)

Below is the server information

Weblogic server version: 10.3.6

OS: Rhel 5.6 64 bit

Below is the script which i am using for starting the server instance

def startWLServer(serverName):

  try:

  start(serverName,'Server')

  print bcolors.BOLD+bcolors.INFO+"<Info> "+serverName+" started successfully"+bcolors.ENDC

  while True:

               redirect('/dev/null','false')

                domainRuntime()

                 slrBean = cmo.lookupServerLifeCycleRuntime(serverName)

               status = slrBean.getState()

                redirect("/dev/null","true")

                 if status.find("RUNNING") != 1:

                       start(serverName,'Server')

                 else:

                      break;

  except:

       dumpStack()

Please let me know.

amey g

Hello,

Is it intermittent? What if you start using node manager or startManagedserver.sh?

Thanks

Amey

amey g

Hello,

Also provide the logs for that particular server at time of issue.

Thanks

Amey

2610779

Below is the logs which i got while running the wlst script for restarting managed server instances.

Starting server mserver ....Server 'mserver' has already been started

No stack trace available.

This Exception occurred at Tue Apr 05 00:35:50 IST 2016.

weblogic.management.scripting.ScriptException: Error occured while performing start : Server with name mserver failed to be started

        at weblogic.management.scripting.ExceptionHandler.handleException(ExceptionHandler.java:48)

        at weblogic.management.scripting.LifeCycleHandler.startServer(LifeCycleHandler.java:502)

        at weblogic.management.scripting.WLScriptContext.start(WLScriptContext.java:523)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.python.core.PyReflectedFunction.__call__(Unknown Source)

        at org.python.core.PyMethod.__call__(Unknown Source)

        at org.python.core.PyObject.__call__(Unknown Source)

        at org.python.core.PyObject.invoke(Unknown Source)

        at org.python.pycode._pyx186.start$80(<iostream>:1272)

        at org.python.pycode._pyx186.call_function(<iostream>)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyFunction.__call__(Unknown Source)

        at org.python.pycode._pyx185.startWLServer$35(/home/wcpcluster/nithin/working/scripts/incDeploy.py:830)

        at org.python.pycode._pyx185.call_function(/home/wcpcluster/nithin/working/scripts/incDeploy.py)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyFunction.__call__(Unknown Source)

        at org.python.pycode._pyx185.f$0(/home/wcpcluster/nithin/working/scripts/incDeploy.py:922)

        at org.python.pycode._pyx185.call_function(/home/wcpcluster/nithin/working/scripts/incDeploy.py)

        at org.python.core.PyTableCode.call(Unknown Source)

        at org.python.core.PyCode.call(Unknown Source)

        at org.python.core.Py.runCode(Unknown Source)

        at org.python.util.PythonInterpreter.execfile(Unknown Source)

        at weblogic.management.scripting.WLST.main(WLST.java:124)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at weblogic.WLST.main(WLST.java:29)

And below is the server logs which i captured at the moment the start() command for starting the managed server instance.

INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.

<Apr 5, 2016 12:35:34 AM IST> <Warning> <oracle.adf.share.jndi.ReferenceStoreHelper> <BEA-000000> <Incomplete connection reference object for connection:ucm>

<Apr 5, 2016 12:35:34 AM IST> <Warning> <oracle.webcenter.worklist.config> <WCS-32009> <The Worklist service does not have a ConnectionName configuration entry in adf-config.xml that maps to a BPELConnection in connections.xml, therefore the Worklist service was not configured for this application.>

<Apr 5, 2016 12:35:34 AM IST> <Warning> <oracle.webcenter.notification.internal.spi.NotificationSenderFactory> <WCS-50027> <Notification sender is not configured>

<Apr 5, 2016 12:35:34 AM IST> <Warning> <oracle.adf.share.ADFContext> <BEA-000000> <Automatically initializing a DefaultContext for getCurrent.

Caller should ensure that a DefaultContext is proper for this use.

Memory leaks and/or unexpected behaviour may occur if the automatic initialization is performed improperly.

This message may be avoided by performing initADFContext before using getCurrent().

To see the stack trace for thread that is initializing this, set the logging level of oracle.adf.share.ADFContext to FINEST>

<Apr 5, 2016 12:35:45 AM IST> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic>

<Apr 5, 2016 12:35:45 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING>

<Apr 5, 2016 12:35:45 AM IST> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>

<Apr 5, 2016 12:35:45 AM IST> <Notice> <Cluster> <BEA-000163> <Stopping "async" replication service>

<Apr 5, 2016 12:35:45 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>

<Apr 5, 2016 12:35:46 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN>

<Apr 5, 2016 12:35:46 AM IST> <Notice> <Server> <BEA-002607> <Channel "Default" listening on <IP_ADDR>:<PORT> was shutdown.>

<Apr 5, 2016 12:35:49 AM IST> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Tue Apr 05 00:35:42 IST 2016) was allocated for an undo, get, or activate operation, but has not been garbage collected.>

<Apr 5, 2016 12:35:49 AM> <FINEST> <NodeManager> <Waiting for the process to die: 27347>

<Apr 5, 2016 12:35:49 AM> <INFO> <NodeManager> <Server was shut down normally>

<Apr 5, 2016 12:35:49 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters

Kindly please let me know, thanks.