4 Replies Latest reply on Oct 12, 2012 2:12 PM by Kent Pawar.

    WebLogic Admin server fails to start after reboot

    Kent Pawar.
      Hi Gurus,

      Weblogic Admin server won't start up after a server reboot. Note that weblogic was gracefully shutdown before the restart.

      Also note that the following files were accidentally deleted after the reboot (but I don't think that should cause an issue..):
      /opt/shared/apps/hyperion/Oracle/Middleware/user_projects/domains/EPMSystem/servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
      /opt/shared/apps/hyperion/Oracle/Middleware/user_projects/domains/EPMSystem/servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT


      Looking at the AdminServer.log:


      *####<Oct 11, 2012 11:27:53 AM EDT> <Critical> <WebLogicServer> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969273846> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: weblogic.management.provider.EditFailedException:*
      java.lang.AssertionError: weblogic.management.provider.EditFailedException:
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:79)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.getInstance(DomainRuntimeServiceMBeanImpl.java:68)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.getDomainConfiguration(DomainRuntimeServiceMBeanImpl.java:132)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.initAppDeploymentRuntimes(DeploymentManagerImpl.java:239)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.<init>(DeploymentManagerImpl.java:83)
           at weblogic.management.deploy.internal.DeploymentServerService.initializeDeploymentManager(DeploymentServerService.java:256)
           at weblogic.management.deploy.internal.DeploymentServerService.startAdminServerDeploymentService(DeploymentServerService.java:233)
           at weblogic.management.deploy.internal.DeploymentServerService.init(DeploymentServerService.java:155)
           at weblogic.management.deploy.internal.DeploymentPreStandbyServerService.start(DeploymentPreStandbyServerService.java:27)
           at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
      Caused By: weblogic.management.provider.EditFailedException:
           at weblogic.management.provider.internal.EditAccessImpl.ensureBeanTreeLoaded(EditAccessImpl.java:1754)
           at weblogic.management.provider.internal.EditAccessImpl.getCurrentDomainBean(EditAccessImpl.java:241)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:76)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.getInstance(DomainRuntimeServiceMBeanImpl.java:68)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.getDomainConfiguration(DomainRuntimeServiceMBeanImpl.java:132)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.initAppDeploymentRuntimes(DeploymentManagerImpl.java:239)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.<init>(DeploymentManagerImpl.java:83)
           at weblogic.management.deploy.internal.DeploymentServerService.initializeDeploymentManager(DeploymentServerService.java:256)
           at weblogic.management.deploy.internal.DeploymentServerService.startAdminServerDeploymentService(DeploymentServerService.java:233)
           at weblogic.management.deploy.internal.DeploymentServerService.init(DeploymentServerService.java:155)
           at weblogic.management.deploy.internal.DeploymentPreStandbyServerService.start(DeploymentPreStandbyServerService.java:27)
           at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
      Caused By: org.xml.sax.SAXParseException: Premature end of file.
           at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)
           at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:174)
           at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:388)
           at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1414)
           at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1059)
           at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
           at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
           at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
           at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
           at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
           at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
           at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:235)
           at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
           at weblogic.xml.jaxp.RegistryDocumentBuilder.parse(RegistryDocumentBuilder.java:163)
           at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:124)
           at weblogic.management.provider.internal.ConfigReader.convert(ConfigReader.java:106)
           at weblogic.management.provider.internal.ConfigReader.<init>(ConfigReader.java:70)
           at weblogic.management.provider.internal.ConfigReader.<init>(ConfigReader.java:64)
           at weblogic.management.provider.internal.EditAccessImpl.loadBeanTree(EditAccessImpl.java:1917)
           at weblogic.management.provider.internal.EditAccessImpl.loadBeanTreeFromPending(EditAccessImpl.java:1807)
           at weblogic.management.provider.internal.EditAccessImpl.ensureBeanTreeLoaded(EditAccessImpl.java:1740)
           at weblogic.management.provider.internal.EditAccessImpl.getCurrentDomainBean(EditAccessImpl.java:241)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:76)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.getInstance(DomainRuntimeServiceMBeanImpl.java:68)
           at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.getDomainConfiguration(DomainRuntimeServiceMBeanImpl.java:132)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.initAppDeploymentRuntimes(DeploymentManagerImpl.java:239)
           at weblogic.management.deploy.internal.DeploymentManagerImpl.<init>(DeploymentManagerImpl.java:81)
           at weblogic.management.deploy.internal.DeploymentServerService.initializeDeploymentManager(DeploymentServerService.java:256)
           at weblogic.management.deploy.internal.DeploymentServerService.startAdminServerDeploymentService(DeploymentServerService.java:232)
           at weblogic.management.deploy.internal.DeploymentServerService.init(DeploymentServerService.java:154)
           at weblogic.management.deploy.internal.DeploymentPreStandbyServerService.start(DeploymentPreStandbyServerService.java:26)
           at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)


      ####<Oct 11, 2012 11:27:54 AM EDT> <Notice> <WebLogicServer> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274051> <BEA-000365> <Server state changed to FAILED>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Error> <WebLogicServer> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274053> <BEA-000383> <A critical service failed. The server will shut itself down>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Notice> <WebLogicServer> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274069> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMX> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274138> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://3.56.11.196:7001/jndi/weblogic.management.mbeanservers.domainruntime .>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMX> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274140> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://3.56.11.196:7001/jndi/weblogic.management.mbeanservers.edit .>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMX> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274141> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://3.56.11.196:7001/jndi/weblogic.management.mbeanservers.runtime .>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <WebService> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274146> <BEA-220028> <Web Service reliable agents are suspended.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <WebService> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274153> <BEA-220029> <Web Service reliable agents are shut down.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <MessagingBridge> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274318> <BEA-200001> <The messaging bridge service has successfully shut down.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMS> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274426> <BEA-040308> <JMS service is suspending.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMS> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274506> <BEA-040107> <Undeployed 8 default connection factories.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JMS> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274510> <BEA-040015> <JMS shutdown is complete.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JDBC> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274530> <BEA-001144> <Force Suspending the JDBC service.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JDBC> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274536> <BEA-001146> <Force suspend of the JDBC service completed.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JDBC> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274538> <BEA-001147> <Shutting down the JDBC service.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <JDBC> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274552> <BEA-001149> <Shutdown of the JDBC service completed.>
      ####<Oct 11, 2012 11:27:54 AM EDT> <Info> <WebLogicServer> <tnsq08041.energy.ge.com> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1349969274694> <BEA-000236> <Stopping execute threads.>
      .....


      Kindly guide me with this. Thanks!

      Edited by: Kent Pawar. on Oct 11, 2012 8:51 AM
        • 1. Re: WebLogic Admin server fails to start after reboot
          sandeep_singh
          Issue is :

          Caused By: weblogic.management.provider.EditFailedException:

          One probable reason can be some change made to the config.xml file . If you have previous backup for the config.xml file then try to replace the config.xml file and restart the server .
          Another reason for this exception may be that there were some changes made to configuration through the Admin Console which were not activated completely and Admin Server was restarted . under this scenario weblogic will create a /pendig dir and have a config.xml file created there with the un-activated changes .

          some workaround you can try to restore the Admin Server :

          Under domain dir ,
          \pending directory -- take backup and delete the dir .

          Under Admin Server Dir :
          data\store --- take backup and delete the dir .

          thanks,
          Sandeep
          • 2. Re: WebLogic Admin server fails to start after reboot
            Kalyan Pasupuleti-Oracle
            Hi Kent,

            As Sandeep mentioned there would be some sort of incorrect order of element is exists.

            Check following XSD file http://www.oracle.com/webfolder/technetwork/weblogic/wls_10.3.3.0.html for Oracle WebLogic Server 10.3.3.0 schema (http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd) .


            You need check which element you make different and change directly with in config.xml file and clear cache and tmp folder and restart the server that should sort your problem.

            Regards,
            Kal
            1 person found this helpful
            • 3. Re: WebLogic Admin server fails to start after reboot
              Kent Pawar.
              Hi Sandeep,

              Thanks that worked for me!

              These are the steps I followed:

              1) Stop the Admin server
              2) Backup the config.xml file, then replace the config.xml file with earlier working config.xml
              Located at: <Domain home>\config\config.xml
              3)Back up the <Domain_Home>/pending and delete the pending directory
              4) Back up temp and cache directory for the server and delete the directories. You may rename these directories.
              5) Back up if any .lok files exist and remove all the .lok files

              I then restarted the server and it worked fine.


              Regards,
              Kent
              • 4. Re: WebLogic Admin server fails to start after reboot
                Kent Pawar.
                Hi Kalyan,

                Your tip to clear cache and tmp folder and restart the server was helpful...
                Also, the config.xml was valid (well-formed) - but I needed to removed the <server-debug>.. attributes that were for some other Managed servers..

                Thanks,
                Kent