This discussion is archived
4 Replies Latest reply: Jan 25, 2013 1:28 PM by 859140 RSS

Weblogic managed server shutdown immediately after going to STARTING state

859140 Newbie
Currently Being Moderated
Our configuration has 3 managed servers M1, M2, M3. M2 and M3 are in a cluster to share JNDI name for JMS servers. Are M3 server often shutdown on starting up just after the server goes to the STARTING state. I have 2 log traces. The first shows when the server shuts down, the 2nd is when the server initializes correctly. No configuration changes, just a timing race condition. Any help is greatly appreciated.....

Bad startup log trace....
Starting WLS with line:
/Oracle/Middleware/jdk1.6.0_25/bin/java -server -Xms2048m -Xmx2048m -XX:NewSizeThreadIncrease=200 -Dweblogic.ThreadPoolPercentSocketReaders=50 -XX:MaxPermSize=512m -Dweblogic.Name=staAdapter -Djava.security.policy=/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/Oracle/Middleware/wlserver_10.3/server/lib/cacerts -da -Dplatform.home=/Oracle/Middleware/wlserver_10.3 -Dwls.home=/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/Oracle/Middleware/wlserver_10.3/server -Ddomain.home=/Oracle/Middleware/user_projects/domains/TBI -Dcommon.components.home=/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig -Doracle.server.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/servers/staAdapter -Doracle.security.jps.config=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/carml -Digf.arisidstack.home=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=false -Dweblogic.management.server=t3://localhost:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Duser.timezone=UTC -Xrs -Dweblogic.MaxMessageSize=100000000 -Dweblogic.ext.dirs=/Oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath:/Oracle/Middleware/patch_jdev1112/profiles/default/sysext_manifest_classpath -Djps.app.credential.overwrite.allowed=true weblogic.Server
<Jan 24, 2013 9:29:16 PM UTC> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 24, 2013 9:29:17 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Jan 24, 2013 9:29:17 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.0-b11 from Sun Microsystems Inc.>
<Jan 24, 2013 9:29:23 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<Jan 24, 2013 9:29:24 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jan 24, 2013 9:29:24 PM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Jan 24, 2013 9:29:24 PM UTC> <Notice> <Log Management> <BEA-170019> <The server log file /Oracle/Middleware/user_projects/domains/TBI/servers/staAdapter/logs/staAdapter.log is opened. All server side log events will be written to this file.>
<Jan 24, 2013 9:29:25 PM UTC> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that libmuxer library is in :'/Oracle/Middleware/jdk1.6.0_25/jre/lib/amd64/server:/Oracle/Middleware/jdk1.6.0_25/jre/lib/amd64:/Oracle/Middleware/jdk1.6.0_25/jre/../lib/amd64:/Oracle/Middleware/patch_wls1035/profiles/default/native:/Oracle/Middleware/patch_jdev1112/profiles/default/native:/Oracle/Middleware/wlserver_10.3/server/native/linux/i686:/Oracle/Middleware/wlserver_10.3/server/native/linux/i686/oci920_8:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib'
>
<Jan 24, 2013 9:29:28 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jan 24, 2013 9:29:30 PM UTC> <Warning> <Store> <BEA-280101> <The persistent file store "_WLS_staAdapter" is forced to use buffered I/O and so may have significantly degraded performance. Either the OS/hardware environment does not support the chosen write policy or the native wlfileio library is missing. See store open log messages for the requested and final write policies. See the documentation on store synchronous write policy configuration for advice.>
<Jan 24, 2013 9:29:31 PM UTC> <Warning> <Deployer> <BEA-149617> <Non-critical internal application uddi was not deployed. Error: [Deployer:149158]No application files exist at '/Oracle/Middleware/wlserver_10.3/server/lib/uddi.war'.>
<Jan 24, 2013 9:29:31 PM UTC> <Warning> <Deployer> <BEA-149617> <Non-critical internal application uddiexplorer was not deployed. Error: [Deployer:149158]No application files exist at '/Oracle/Middleware/wlserver_10.3/server/lib/uddiexplorer.war'.>
<Jan 24, 2013 9:29:31 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Jan 24, 2013 9:29:31 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
*<Jan 24, 2013 9:29:51 PM UTC> <Notice> <WebLogicServer> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>*
*<Jan 24, 2013 9:29:51 PM UTC> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>*
*<Jan 24, 2013 9:29:51 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>*


Good startup trace.....
Starting WLS with line:
/Oracle/Middleware/jdk1.6.0_25/bin/java -server -Xms2048m -Xmx2048m -XX:NewSizeThreadIncrease=200 -Dweblogic.ThreadPoolPercentSocketReaders=50 -XX:MaxPermSize=512m -Dweblogic.Name=staAdapter -Djava.security.policy=/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/Oracle/Middleware/wlserver_10.3/server/lib/cacerts -da -Dplatform.home=/Oracle/Middleware/wlserver_10.3 -Dwls.home=/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/Oracle/Middleware/wlserver_10.3/server -Ddomain.home=/Oracle/Middleware/user_projects/domains/TBI -Dcommon.components.home=/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig -Doracle.server.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/servers/staAdapter -Doracle.security.jps.config=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/carml -Digf.arisidstack.home=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=false -Dweblogic.management.server=t3://localhost:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Duser.timezone=UTC -Xrs -Dweblogic.MaxMessageSize=100000000 -Dweblogic.ext.dirs=/Oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath:/Oracle/Middleware/patch_jdev1112/profiles/default/sysext_manifest_classpath -Djps.app.credential.overwrite.allowed=true weblogic.Server
<Jan 24, 2013 5:31:06 PM UTC> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 24, 2013 5:31:06 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Jan 24, 2013 5:31:06 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.0-b11 from Sun Microsystems Inc.>
<Jan 24, 2013 5:31:07 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<Jan 24, 2013 5:31:08 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jan 24, 2013 5:31:08 PM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Jan 24, 2013 5:31:08 PM UTC> <Notice> <Log Management> <BEA-170019> <The server log file /Oracle/Middleware/user_projects/domains/TBI/servers/staAdapter/logs/staAdapter.log is opened. All server side log events will be written to this file.>
<Jan 24, 2013 5:31:09 PM UTC> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that libmuxer library is in :'/Oracle/Middleware/jdk1.6.0_25/jre/lib/amd64/server:/Oracle/Middleware/jdk1.6.0_25/jre/lib/amd64:/Oracle/Middleware/jdk1.6.0_25/jre/../lib/amd64:/Oracle/Middleware/patch_wls1035/profiles/default/native:/Oracle/Middleware/patch_jdev1112/profiles/default/native:/Oracle/Middleware/wlserver_10.3/server/native/linux/i686:/Oracle/Middleware/wlserver_10.3/server/native/linux/i686/oci920_8:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib'
>
<Jan 24, 2013 5:31:13 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jan 24, 2013 5:31:14 PM UTC> <Warning> <Store> <BEA-280101> <The persistent file store "_WLS_staAdapter" is forced to use buffered I/O and so may have significantly degraded performance. Either the OS/hardware environment does not support the chosen write policy or the native wlfileio library is missing. See store open log messages for the requested and final write policies. See the documentation on store synchronous write policy configuration for advice.>
<Jan 24, 2013 5:31:15 PM UTC> <Warning> <Deployer> <BEA-149617> <Non-critical internal application uddi was not deployed. Error: [Deployer:149158]No application files exist at '/Oracle/Middleware/wlserver_10.3/server/lib/uddi.war'.>
<Jan 24, 2013 5:31:15 PM UTC> <Warning> <Deployer> <BEA-149617> <Non-critical internal application uddiexplorer was not deployed. Error: [Deployer:149158]No application files exist at '/Oracle/Middleware/wlserver_10.3/server/lib/uddiexplorer.war'.>
<Jan 24, 2013 5:31:16 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Jan 24, 2013 5:31:16 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
**<Jan 24, 2013 5:31:30 PM UTC> <Notice> <Log Management> <BEA-170027> <The Server has established connection with the Domain level Diagnostic Service successfully.>**
**<Jan 24, 2013 5:31:30 PM UTC> <Notice> <Cluster> <BEA-000197> <Listening for announcements from cluster using unicast cluster messaging>**
**<Jan 24, 2013 5:31:30 PM UTC> <Notice> <Cluster> <BEA-000133> <Waiting to synchronize with other running members of STA_cluster.>**
**<Jan 24, 2013 5:31:38 PM UTC> <Notice> <Cluster> <BEA-000142> <Trying to download cluster JNDI tree from server staEngine.>**
**<Jan 24, 2013 5:31:38 PM UTC> <Notice> <Cluster> <BEA-000164> <Synchronized cluster JNDI tree from server staEngine.>**
**<Jan 24, 2013 5:31:38 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>**
**<Jan 24, 2013 5:31:38 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>**
  • 1. Re: Weblogic managed server shutdown immediately after going to STARTING state
    Kishore Rout Journeyer
    Currently Being Moderated
    Hi,
    This problem happens when an erroneous shutdown signal is sent by a component to the JVM. The JVM catches those signals, and shuts down all Java processes.
    SJVM catches signals to implement shutdown hooks for abnormal JVM termination.

    Solution to this problem:
    Back up DOMAIN_HOME\bin\setDomainEnv.sh file.
    edit DOMAIN_HOME/bin/setDomainEnv.sh and change

    From:
    JAVA_OPTIONS="${JAVA_OPTIONS}"
    export JAVA_OPTIONS
    To:
    JAVA_OPTIONS="${JAVA_OPTIONS} -Xrs"
    export JAVA_OPTIONS

    mark if this helps.

    Regads,
    Kishore
  • 2. Re: Weblogic managed server shutdown immediately after going to STARTING state
    859140 Newbie
    Currently Being Moderated
    Thanks. Unfortunately I've already done that and it doesn't work. I've included the -Xrs option in the managed server java command line arguments, see below. However, either it is in the wrong location on the java argument list or something else is going on, because it is not working. See the JVM command line below of the managed server.....

    /Oracle/Middleware/jdk1.6.0_25/bin/java -server -Xms2048m -Xmx2048m -XX:NewSizeThreadIncrease=200 -Dweblogic.ThreadPoolPercentSocketReaders=50 -XX:MaxPermSize=512m -Dweblogic.Name=staAdapter -Djava.security.policy=/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/Oracle/Middleware/wlserver_10.3/server/lib/cacerts -da -Dplatform.home=/Oracle/Middleware/wlserver_10.3 -Dwls.home=/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/Oracle/Middleware/wlserver_10.3/server -Ddomain.home=/Oracle/Middleware/user_projects/domains/TBI -Dcommon.components.home=/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig -Doracle.server.config.dir=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/servers/staAdapter -Doracle.security.jps.config=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/carml -Digf.arisidstack.home=/Oracle/Middleware/user_projects/domains/TBI/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=false -Dweblogic.management.server=t3://localhost:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Duser.timezone=UTC -Xrs -Dweblogic.MaxMessageSize=100000000 -Dweblogic.ext.dirs=/Oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath:/Oracle/Middleware/patch_jdev1112/profiles/default/sysext_manifest_classpath -Djps.app.credential.overwrite.allowed=true weblogic.Serve
  • 3. Re: Weblogic managed server shutdown immediately after going to STARTING state
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Hi,

    Mostly hook can avoid by using -Xrs command you can try to clear tmp and cache folder for specific server and try to restart and check whether you see similar issue or not.

    Note : when you set -Xrs we cannot take thread dump using kill -3 option.


    Regards,
    Kal
  • 4. Re: Weblogic managed server shutdown immediately after going to STARTING state
    859140 Newbie
    Currently Being Moderated
    Found the issue. Turns out that EJBs are initialized immediately after the MS goes to STARTING health state. There was a static variable in someone's EJB that was used on 2 different managed servers. The JVM was throwing an exception that was unfortunately not being recorded on any of the WL logs. After re-designing the EJB, the JVM no long causes the WL shutdown.

    We also found a caching issue, which were are going to try to use the suggestion from kalyan Pasupuleti to fix.

    Thanks,
    Mark M.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points