1 Reply Latest reply on Sep 26, 2016 9:54 AM by Ssukhija-Oracle

    adadminsrvctl.sh is failing during cutover phase

    Girish Sunkara

      During the cutover phase, starting the new RUN file system is failing with status 1. In logs it is showing that admin server is failing to start.

       

       

      Below are contents of adadminsrvctl log file,

       

       

      stty: standard input: Inappropriate ioctl for device

      Enter the WebLogic AdminServer password:

      stty: standard input: Inappropriate ioctl for device

      The log file is /u01/apps/fs1/inst/apps/PROD_ebs122/logs/appl/rgf/Mon_Sep_12_20_23_47_2016/adRegisterWLSListeners.log

       

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is in RUNNING mode.

       

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is in RUNNING mode.

       

      09/13/16-03:36:22 :: adadminsrvctl.sh version 120.10.12020000.2

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is in RUNNING mode.

       

      Validated the passed arguments for the option ebs-nmstop-adminsrv

      Connecting to t3://ebs122.vishnudba.com:7001 with userid weblogic ...

      Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_PROD'.

       

      Warning: An insecure protocol was used to connect to the

      server. To ensure on-the-wire security, the SSL port or

      Admin port should be used instead.

       

      Sep 13, 2016 3:36:51 AM ClientCommunicatorAdmin restart

      WARNING: Failed to restart: java.rmi.NoSuchObjectException: The object identified by: '283' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.

      Sep 13, 2016 3:36:51 AM ClientCommunicatorAdmin restart

      WARNING: Failed to restart: java.rmi.NoSuchObjectException: The object identified by: '284' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.

       

      Shutting down the server AdminServer with force=true while connected to AdminServer ...

      Disconnected from weblogic server: AdminServer

      ================================================================================

      09/20/16-20:33:12 :: adadminsrvctl.sh version 120.10.12020000.2

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is currently not running.

       

      Validated the passed arguments for the option ebs-nmstart-adminsrv

      Checking if the Admin Server is already up.

      The Admin Server is not already up.

      Traceback (innermost last):

        File "<string>", line 1, in ?

        File "<iostream>", line 123, in nmConnect

        File "<iostream>", line 656, in raiseWLSTException

      WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at ebs122.vishnudba.com:5556.

      Use dumpStack() to view the full stacktrace

      09/13/16-03:36:52 :: adadminsrvctl.sh: exiting with status 0

       

      ================================================================================

       

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is currently not running.

       

      09/19/16-15:35:54 :: adadminsrvctl.sh version 120.10.12020000.2

      Validated the passed arguments for the option ebs-get-serverstatus

      AdminServer is currently not running.

       

      09/19/16-15:36:03 :: adadminsrvctl.sh: exiting with status 2

       

       

       

      Below are the contents of AdminServer.out

       

      WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >

      <Sep 23, 2016 7:28:58 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

      <Sep 23, 2016 7:28:58 AM EDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>

      <Sep 23, 2016 7:28:59 AM EDT> <Notice> <Log Management> <BEA-170019> <The server log file /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>

      Sep 23, 2016 7:29:13 AM oracle.security.jps.internal.idstore.util.LibOvdUtil pushLdapNamesToLibOvd

      INFO: Pushed ldap name and types info to libOvd. Ldaps : DefaultAuthenticator:idstore.ldap.provideridstore.ldap.

      <Sep 23, 2016 7:29:17 AM EDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>

      Sep 23, 2016 7:29:23 AM oracle.security.jps.JpsStartup start

      INFO: Jps initializing.

      Sep 23, 2016 7:29:23 AM oracle.security.jps.JpsStartup start

      INFO: Jps started.

      <Sep 23, 2016 7:29:28 AM EDT> <Warning> <oracle.as.jmx.framework.generic.spi.BaseJMXFrameworkProviderImpl> <BEA-000000> <The configuration at URI "/u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/mbeans/../default-keystore.jks" cannot be loaded.>

      <Sep 23, 2016 7:29:29 AM EDT> <Error> <Management> <BEA-141244> <Schema validation errors while parsing /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/config/config.xml - Expected elements 'alt-descriptor-path@http://xmlns.oracle.com/weblogic/domain alt-wls-descriptor-path@http://xmlns.oracle.com/weblogic/domain application-identifier@http://xmlns.oracle.com/weblogic/domain application-name@http://xmlns.oracle.com/weblogic/domain' instead of 'cache-in-app-directory@http://xmlns.oracle.com/weblogic/domain' here in element app-deployment@http://xmlns.oracle.com/weblogic/domain>

      <Sep 23, 2016 7:29:29 AM EDT> <Error> <Management> <BEA-141244> <Schema validation errors while parsing /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/config/config.xml - Expected elements 'alt-descriptor-path@http://xmlns.oracle.com/weblogic/domain alt-wls-descriptor-path@http://xmlns.oracle.com/weblogic/domain application-identifier@http://xmlns.oracle.com/weblogic/domain application-name@http://xmlns.oracle.com/weblogic/domain' instead of 'cache-in-app-directory@http://xmlns.oracle.com/weblogic/domain' here in element library@http://xmlns.oracle.com/weblogic/domain>

      <Sep 23, 2016 7:29:29 AM EDT> <Error> <Management> <BEA-141244> <Schema validation errors while parsing /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/config/config.xml - Expected elements 'alt-descriptor-path@http://xmlns.oracle.com/weblogic/domain alt-wls-descriptor-path@http://xmlns.oracle.com/weblogic/domain application-identifier@http://xmlns.oracle.com/weblogic/domain application-name@http://xmlns.oracle.com/weblogic/domain' instead of 'cache-in-app-directory@http://xmlns.oracle.com/weblogic/domain' here in element library@http://xmlns.oracle.com/weblogic/domain>

      <Sep 23, 2016 7:29:29 AM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: weblogic.management.provider.EditFailedException:

      java.lang.AssertionError: weblogic.management.provider.EditFailedException:

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:79)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.getInstance(DomainRuntimeServiceMBeanImpl.java:68)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.getDomainConfiguration(DomainRuntimeServiceMBeanImpl.java:132)

              at weblogic.management.deploy.internal.DeploymentManagerImpl.initAppDeploymentRuntimes(DeploymentManagerImpl.java:239)

              at weblogic.management.deploy.internal.DeploymentManagerImpl.<init>(DeploymentManagerImpl.java:81)

              Truncated. see log file for complete stacktrace

      Caused By: weblogic.management.provider.EditFailedException:

              at weblogic.management.provider.internal.EditAccessImpl.ensureBeanTreeLoaded(EditAccessImpl.java:1844)

              at weblogic.management.provider.internal.EditAccessImpl.getCurrentDomainBean(EditAccessImpl.java:241)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:76)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.getInstance(DomainRuntimeServiceMBeanImpl.java:68)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.getDomainConfiguration(DomainRuntimeServiceMBeanImpl.java:132)

              Truncated. see log file for complete stacktrace

      Caused By: java.io.IOException: [Management:141245]Schema Validation Error in /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/config/config.xml see log for details. Schema validation can be disabled by starting the server with the command line option: -Dweblogic.configuration.schemaValidationEnabled=false

              at weblogic.management.provider.internal.EditAccessImpl.checkErrors(EditAccessImpl.java:2340)

              at weblogic.management.provider.internal.EditAccessImpl.loadBeanTreeFromActive(EditAccessImpl.java:1863)

              at weblogic.management.provider.internal.EditAccessImpl.ensureBeanTreeLoaded(EditAccessImpl.java:1830)

              at weblogic.management.provider.internal.EditAccessImpl.getCurrentDomainBean(EditAccessImpl.java:241)

              at weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl$SINGLETON.<init>(DomainRuntimeServiceMBeanImpl.java:76)

              Truncated. see log file for complete stacktrace

      >

      <Sep 23, 2016 7:29:29 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

      <Sep 23, 2016 7:29:29 AM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

      <Sep 23, 2016 7:29:29 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

      <Sep 23, 2016 7:29:30 AM> <FINEST> <NodeManager> <Waiting for the process to die: 28141>

      <Sep 23, 2016 7:29:30 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

      <Sep 23, 2016 7:29:30 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>