0 Replies Latest reply: Mar 7, 2013 11:54 PM by Krishna1 RSS

    Not able to start the managed servers throughing "Failed-not restartable "

    Krishna1
      Hello All...

      We have installed New weblogic portal 10.3.5 on solaris sparc server
      We have used Jrockit 1.6.37 version.
      We have installed Derby default database for portal.


      We are able to browse the admin console and able to see two managed nodes.

      Now we are not able to start the managed nodes.
      We received error “Failed-not restartable” in Admin console page for managed node.

      We have created nodemanager in multicast.

      Please find below the managed node error logs for your reference.

      <Error> <NodeManager> <BEA-300048> <Unable to start the server wl-qmtctx-qmt-1 : Exception while starting server 'wl-qmtctx-qmt-1'>
      <Error> <NodeManager> <BEA-300048> <Unable to start the server wl-qmtctx-qmt-1 : Exception while starting server 'wl-qmtctx-qmt-1'>
      Feb 28, 2013 10:46:18 AM MET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
      <Feb 28, 2013 10:46:18 AM MET> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
      <Feb 28, 2013 10:46:18 AM MET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
      <Feb 28, 2013 10:46:19 AM> <FINEST> <NodeManager> <Waiting for the process to die: 29186>
      <Feb 28, 2013 10:46:19 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
      <Feb 28, 2013 10:46:19 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
      <Feb 28, 2013 10:46:17 AM MET> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

      There are 1 nested errors:

      weblogic.security.service.SecurityServiceRuntimeException: [Security:090399]Security Services Unavailable
      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      <Feb 28, 2013 10:46:17 AM MET> <Error> <Security> <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException:
      com.bea.common.engine.ServiceInitializationException: <openjpa-1.1.1-SNAPSHOT-r422266:965591 fatal internal error> org.apache.openjpa.util.InternalException:
      There was an error when invoking the static getInstance method on the named factory class "kodo.jdbc.kernel.KodoJDBCBrokerFactory". See the nested exceptio
      n for details..
      weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: <openjpa-1.1.1-SNAPSHOT-r422266:965591 fatal intern
      al error> org.apache.openjpa.util.InternalException: There was an error when invoking the static getInstance method on the named factory class "kodo.jdbc.ker
      nel.KodoJDBCBrokerFactory". See the nested exception for details.
      at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:341)
      at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:220)
      at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1785)
      at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:442)
      at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:840)
      Truncated. see log file for complete stacktrace
      Caused By: com.bea.common.engine.ServiceInitializationException: <openjpa-1.1.1-SNAPSHOT-r422266:965591 fatal internal error> org.apache.openjpa.util.Interna
      lException: There was an error when invoking the static getInstance method on the named factory class "kodo.jdbc.kernel.KodoJDBCBrokerFactory". See the nest
      ed exception for details.
      at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
      at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
      at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
      at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
      :$
      >
      <Feb 28, 2013 10:46:17 AM MET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
      <Feb 28, 2013 10:46:17 AM MET> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

      There are 1 nested errors:

      weblogic.security.service.SecurityServiceRuntimeException: [Security:090399]Security Services Unavailable
      at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:916)
      at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050)
      at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
      at weblogic.security.SecurityService.start(SecurityService.java:141)
      at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

      >
      <Feb 28, 2013 10:46:18 AM MET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
      <Feb 28, 2013 10:46:18 AM MET> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
      <Feb 28, 2013 10:46:18 AM MET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
      <Feb 28, 2013 10:46:19 AM> <FINEST> <NodeManager> <Waiting for the process to die: 29186>
      <Feb 28, 2013 10:46:19 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
      <Feb 28, 2013 10:46:19 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
      Nodemanager logs as follows:


      <Feb 28, 2013 10:36:07 AM> <WARNING> <Node manager configuration properties file '/app/Middleware/wlserver_10.3/common/nodemanager/nodemanager.properties' no
      t found. Using default settings.>
      <Feb 28, 2013 10:39:10 AM> <WARNING> <Exception while starting server 'wl-qmtctx-qmt-1'>
      java.io.IOException: Server failed to start up. See server output log for more details.
      at weblogic.nodemanager.server.AbstractServerManager.start(AbstractServerManager.java:200)
      at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:23)
      at weblogic.nodemanager.server.Handler.handleStart(Handler.java:604)
      Admin Server logs:

      ####<Feb 28, 2013 10:34:22 AM MET> <Error> <Security> <esvmzn007> <wl-qmt-admin> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tunin
      g)'> <<WLS Kernel>> <> <59424a3c17cd2a6b:382c951d:13d20260c63:-8000-0000000000000003> <1362044062548> <BEA-000000> <Commit events are not broadcast because n
      o JMS Transport is available. Remote L2 caches will be out of sync.>
      ####<Feb 28, 2013 10:34:22 AM MET> <Error> <Security> <esvmzn007> <wl-qmt-admin> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tunin
      g)'> <<WLS Kernel>> <> <59424a3c17cd2a6b:382c951d:13d20260c63:-8000-0000000000000003> <1362044062713> <BEA-000000> <Commit events are not broadcast because n
      o JMS Transport is available. Remote L2 caches will be out of sync.>

      =============================================================
      we have troubleshoot like below...still not able to start the managed server....please can any one help me

      1) FAILED NOT RESTARTABLE :

      FAILED NOT RESTARTABLE is a common error in the Node Manager environment and This could be because of two reasons
      1) WebLogic Server which you are trying to start is already running.
      2) WebLogic Server which you are trying to start did not stop cleanly.

      When Web Logic server stops, it removes these two files. If you not stop server properly these lock files remains on that location and creating a problem when you attempt to start a server again and after more than one attempt your server goes in FAILED_NOT_RESTARTABLE state.
      i) Domain home/servers/<server_name>/tmp/<servermame>.lok
      ii) Domain home /servers/<server_name>/data/ldap/ldapfiles/EmbeddedLDAP.lok

      and also we have stop the all servers and we stop the servers..same error we are getting...


      Regards..
      Krishna

      Edited by: 988913 on 7 Mar, 2013 9:51 PM