1 2 Previous Next 18 Replies Latest reply on Sep 17, 2014 3:28 PM by TheKop88

    The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>

    1556169

      Logs stuck while starting the weblogic server:


      <Sep 12, 2014 11:20:37 AM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64 from Oracle Corporation>

      <Sep 12, 2014 11:20:39 AM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >

      <Sep 12, 2014 11:20:47 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

      <Sep 12, 2014 11:20:47 AM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>


      When trying to run ./stopWeblogic.sh Stuck at below location:

      *****************************************************

      ** Setting up OIM specific environment...

      JVM Type is java version "1.6.0_37"

      Java(TM) SE Runtime Environment (build 1.6.0_37-b06)

      Oracle JRockit(R) (build R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64, compiled mode)

      .

      USER_MEM_ARGS=-Xms1024m -Xmx2048m

      .

      *****************************************************

      ** End OIM specific environment setup

      *****************************************************

      Stopping Weblogic Server...

      [WARN ] Use of -Djrockit.optfile is deprecated and discouraged.

       

       

      Initializing WebLogic Scripting Tool (WLST) ...

       

       

      Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.

       

       

      *sys-package-mgr*: can't create package cache dir, '/tmpWLSTTempiamadmin/packages'

      java.io.IOException: Read-only file system

              at java.io.UnixFileSystem.createFileExclusively(Native Method)

              at java.io.File.checkAndCreate(File.java:1705)

              at java.io.File.createTempFile0(File.java:1726)

              at java.io.File.createTempFile(File.java:1838)

              at com.oracle.cie.domain.script.jython.WLST_offline.getWLSTOfflineInitFilePath(WLST_offline.java:239)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at weblogic.management.scripting.utils.WLSTUtil.getOfflineWLSTScriptPathInternal(WLSTUtil.java:85)

              at weblogic.management.scripting.utils.WLSTUtil.setupOfflineInternal(WLSTUtil.java:255)

              at weblogic.management.scripting.utils.WLSTUtil.setupOffline(WLSTUtil.java:234)

              at weblogic.management.scripting.utils.WLSTInterpreter.<init>(WLSTInterpreter.java:134)

              at weblogic.management.scripting.WLST.main(WLST.java:98)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at weblogic.WLST.main(WLST.java:29)

      Problem invoking WLST - java.lang.NullPointerException

        • 1. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
          1556169

          Logs when trying to start weblogic server with command ./startWeblogic.sh

           

           

          .

          ***************************************************

          *  To start WebLogic Server, use a username and   *

          *  password assigned to an admin-level user.  For *

          *  server administration, use the WebLogic Server *

          *  console at http://hostname:port/console        *

          ***************************************************

          starting weblogic with Java version:

          java version "1.6.0_37"

          Java(TM) SE Runtime Environment (build 1.6.0_37-b06)

          Oracle JRockit(R) (build R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64, compiled mode)

          Starting WLS with line:

          /u01/Oracle/Middleware/jrockit-jdk1.6.0_26/bin/java -jrockit   -Xms1024m -Xmx2048m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true   -da -Dplatform.home=/u01/Oracle/Middleware/wlserver_10.3 -Dwls.home=/u01/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/u01/Oracle/Middleware/wlserver_10.3/server  -Dcommon.components.home=/u01/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain -Djrockit.optfile=/u01/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig  -Digf.arisidbeans.carmlloc=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/carml  -Digf.arisidstack.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/Oracle/Middleware/Oracle_IDM1/server/loginmodule/wls,/u01/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol  -Dweblogic.jdbc.remoteEnabled=true  -DXL.HomeDir=/u01/Oracle/Middleware/Oracle_IDM1/server  -Djava.security.auth.login.config=/u01/Oracle/Middleware/Oracle_IDM1/server/config/authwl.conf  -Dorg.owasp.esapi.resources=/u01/Oracle/Middleware/Oracle_IDM1/server/apps/oim.ear/APP-INF/classes  -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/Oracle/Middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/Oracle/Middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain -Dtangosol.coherence.clusteraddress=227.7.7.9 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/Oracle/Middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/Oracle/Middleware/oracle_common -Djava.awt.headless=true -Dums.oracle.home=/u01/Oracle/Middleware/Oracle_SOA1 -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/Oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/u01/Oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath  weblogic.Server

          [WARN ] Use of -Djrockit.optfile is deprecated and discouraged.

          <Sep 15, 2014 9:29:34 AM UTC> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>

          <Sep 15, 2014 9:29:35 AM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>

          <Sep 15, 2014 9:29:35 AM UTC> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:

          /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/lib/sqljdbc.jar>

          <Sep 15, 2014 9:29:35 AM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64 from Oracle Corporation>

          <Sep 15, 2014 9:29:37 AM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >

          <Sep 15, 2014 9:29:43 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

          <Sep 15, 2014 9:29:43 AM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool

           

           

          After <Initializing self-tuning thread pool no update.

          tried delting all the file from tmp and cache as well from  location :"Domain home /servers/AdminServer /tmp" but still the issue exist.

          Also tried deleting the lok file from location "Domain home /servers/AdminServer/tmp/AdminServer.lok" but could not solve it.

           

           

          Please help to resolve it. Not able to start the managed server present on Admin server.

          • 2. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
            TheKop88

            Hi ,

            Which user are you using to start/stop WebLogic ? Does that user have permission to read/write  /tmp directory ?
            Try to move Embedded LDAP files from WebLogic Server  and then run ./startWeblogic.sh.
            Embedded LDAP files in WebLogic are stored at $DOMAIN_HOME/servers/[ServerName]/data/ldap.

             

            Thanks  

            • 3. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
              1556169

              Hello ,

               

              Thank you for your reply.

              I am trying to run the start weblogic command with  admin user only and it has read/write access to all the folders.

               

              And regarding Embedded LDAP files ,where shall I try to move it. Is it just that I can rename the file or should I move it to some other location.

               

              Thanks ,

              Nivedita

              • 4. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                TheKop88

                Hi ,

                Yes ,  you can rename ldap folder . One more thing,  I can see
                "sys-package-mgr*: can't create package cache dir, '/tmpWLSTTempiamadmin/packages'

                java.io.IOException: Read-only file system "

                Check if you have read/write permission to "/tmpWLSTTempiamadmin" directory.

                Thanks

                • 5. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                  1556169

                  Hi Tried after renaimng the Folder.

                  Getting below error now. Also could you please let me know the folder structure where I can find directory '/tmpWLSTTempiamadmin/packages'. I tried searching but could not find.

                   

                   

                  hange, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>

                  <Sep 15, 2014 12:49:53 PM UTC> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:

                  /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/lib/sqljdbc.jar>

                  <Sep 15, 2014 12:49:54 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64 from Oracle Corporation>

                  <Sep 15, 2014 12:50:07 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:50:19 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:50:29 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:50:39 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:50:49 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:50:59 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:51:09 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:51:19 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:51:29 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:51:39 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:51:50 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:00 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:10 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:20 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:30 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:40 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:52:50 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:53:00 PM UTC> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>

                  <Sep 15, 2014 12:53:02 PM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

                   

                   

                  There are 1 nested errors:

                   

                   

                  weblogic.management.ManagementException: Unable to obtain lock on /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/tmp/AdminServer.lok. Server may already be running

                          at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:206)

                          at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:67)

                          at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:74)

                          at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)

                          at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)

                          at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:882)

                          at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:572)

                          at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)

                          at weblogic.Server.main(Server.java:71)

                   

                   

                  >

                  <Sep 15, 2014 12:53:02 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

                  <Sep 15, 2014 12:53:02 PM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

                  <Sep 15, 2014 12:53:02 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

                  Calculating line numbers... (interrupt to abort)

                   

                   

                   

                  Thanks

                  Nivedita

                  • 6. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                    TheKop88

                    Hi ,
                    "Unable to obtain lock on /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/tmp/AdminServer.lok."

                    delete this(AdminSevrer.lok) lok file . Kill any processes from previous start of  ./startWeblogic.sh

                    Thanks

                    • 7. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                      1556169

                      Hi,

                      Deleted the lok file and also killed all the process which were running for AdminServer.

                      Once again stuck with the same message

                       

                       

                       

                      ***************************************************

                      *  To start WebLogic Server, use a username and   *

                      *  password assigned to an admin-level user.  For *

                      *  server administration, use the WebLogic Server *

                      *  console at http://hostname:port/console        *

                      ***************************************************

                      starting weblogic with Java version:

                      java version "1.6.0_37"

                      Java(TM) SE Runtime Environment (build 1.6.0_37-b06)

                      Oracle JRockit(R) (build R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64, compiled mode)

                      Starting WLS with line:

                      /u01/Oracle/Middleware/jrockit-jdk1.6.0_26/bin/java -jrockit   -Xms1024m -Xmx2048m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true   -da -Dplatform.home=/u01/Oracle/Middleware/wlserver_10.3 -Dwls.home=/u01/Oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/u01/Oracle/Middleware/wlserver_10.3/server  -Dcommon.components.home=/u01/Oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain -Djrockit.optfile=/u01/Oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig  -Digf.arisidbeans.carmlloc=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/carml  -Digf.arisidstack.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/Oracle/Middleware/Oracle_IDM1/server/loginmodule/wls,/u01/Oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/Oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol  -Dweblogic.jdbc.remoteEnabled=true  -DXL.HomeDir=/u01/Oracle/Middleware/Oracle_IDM1/server  -Djava.security.auth.login.config=/u01/Oracle/Middleware/Oracle_IDM1/server/config/authwl.conf  -Dorg.owasp.esapi.resources=/u01/Oracle/Middleware/Oracle_IDM1/server/apps/oim.ear/APP-INF/classes  -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/Oracle/Middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/Oracle/Middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/Oracle/Admin/IDMDomain/mserver/IDMDomain -Dtangosol.coherence.clusteraddress=227.7.7.9 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/Oracle/Middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/Oracle/Middleware/oracle_common -Djava.awt.headless=true -Dums.oracle.home=/u01/Oracle/Middleware/Oracle_SOA1 -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/Oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/u01/Oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath  weblogic.Server

                      [WARN ] Use of -Djrockit.optfile is deprecated and discouraged.

                      <Sep 15, 2014 1:13:06 PM UTC> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>

                      <Sep 15, 2014 1:13:07 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>

                      <Sep 15, 2014 1:13:07 PM UTC> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:

                      /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/lib/sqljdbc.jar>

                      <Sep 15, 2014 1:13:07 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64 from Oracle Corporation>

                      <Sep 15, 2014 1:13:10 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >

                      <Sep 15, 2014 1:13:17 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

                      <Sep 15, 2014 1:13:17 PM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool

                       

                      Thanks

                      Nivedita

                      • 8. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                        TheKop88

                        Hi ,

                        Are there any log files generated under $DOMAIN_HOME/servers/AdminServer/logs directory ? Something like AdminSevrer.log or AdminServer.out ? Is there any error in these files. ? For how long have you waited before you declared it stuck ?

                        Thanks

                        • 9. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                          Luz Mestre-Oracle

                          Please try collecting 3 thread dumps to determine what is the process stuck about.

                          e.g:

                          kill -3 <PID>

                          wait 10 secs

                          kill -3 <PID>

                          wait 10 secs

                          kill -3 <PID>

                           

                          where PID is the process ID of WLS that is not starting

                          kill -3 will collect information in the standart output

                           

                          To have captured standard output, you can start Admin Server using

                          nohup ./startWeblogic.sh

                          Best Regards

                          Luz

                          • 10. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                            1556169

                            Hi,

                             

                            It is been stuck till now. And the last log file which got generated under $DOMAIN_HOME/servers/AdminServer/logs was of date 10th September where we were able to see an exception :

                             

                            ####<Sep 10, 2014 11:42:47 AM UTC> <Info> <JDBC> <ct11bzapp262.am.mot-solutions.com> <AdminServer> <Policy change scanner> <<anonymous>> <> <0000KWSjnBkEWNJLMmCCye1JzmfD000000> <1410349367492> <BEA-001156> <Stack trace associated with message 001129 follows:

                            java.sql.SQLException: ORA-28001: the password has expired

                                     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:462)
                                     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:397)


                            I have requested for reseting of password of SOA Infra which was expired. But after the DB has rest the password the logs were not generated though  I tried executing the start command after 10th of September.

                            Also want to mention that I have renamed the two folder tmp and cache under  ( /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/tmp/) /u01/Oracle/Admin/IDMDomain/mserver/IDMDomain/servers/AdminServer/cache) as well trying to get the Admin Server started. But till now the issue exist.

                             

                            Kindly Suggest.

                             

                             

                            Thanks,

                            Nivedita

                            • 11. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                              1556169

                              Hello Luz,

                               

                              Please find below the thread dumps collected :

                              For OIM PID

                               

                              -- Waiting for notification on: java/util/TaskQueue@0xa1957470[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at java/util/TimerThread.mainLoop(Timer.java:483)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa1957470[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "Timer-1" id=14 idx=0x38 tid=29297 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: java/util/TaskQueue@0xa1de2240[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/util/TimerThread.mainLoop(Timer.java:509)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa1de2240[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=29298 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: weblogic/work/ExecuteThread@0xa161c2c8[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:205)

                                  ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xa161c2c8[fat lock]

                                  at weblogic/work/ExecuteThread.run(ExecuteThread.java:226)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "JFR request timer" id=16 idx=0x40 tid=29299 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: java/util/TaskQueue@0xa1de22c0[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at java/util/TimerThread.mainLoop(Timer.java:483)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa1de22c0[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              ===== END OF THREAD DUMP ===============

                              Calculating line numbers... (interrupt to abort)

                               

                              For PID for AdminServer

                               

                                  -- Waiting for notification on: java/util/TaskQueue@0xa157ca58[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at java/util/TimerThread.mainLoop(Timer.java:483)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa157ca58[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "Timer-1" id=14 idx=0x38 tid=11774 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: java/util/TaskQueue@0xa157cac0[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/util/TimerThread.mainLoop(Timer.java:509)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa157cac0[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=11775 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: weblogic/work/ExecuteThread@0xa157d4b8[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:205)

                                  ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xa157d4b8[fat lock]

                                  at weblogic/work/ExecuteThread.run(ExecuteThread.java:226)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              "JFR request timer" id=16 idx=0x40 tid=11776 prio=5 alive, waiting, native_blocked, daemon

                                  -- Waiting for notification on: java/util/TaskQueue@0xa157cb40[fat lock]

                                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                                  at java/lang/Object.wait(J)V(Native Method)

                                  at java/lang/Object.wait(Object.java:485)

                                  at java/util/TimerThread.mainLoop(Timer.java:483)

                                  ^-- Lock released while waiting: java/util/TaskQueue@0xa157cb40[fat lock]

                                  at java/util/TimerThread.run(Timer.java:462)

                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                                  -- end of trace

                               

                               

                              ===== END OF THREAD DUMP ===============

                              • 12. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                                1556169

                                More logs from Adminserver.log

                                 

                                 

                                 

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:293)

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1508)

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:579)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1717)

                                        at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:264)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:646)

                                        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2611)

                                        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2570)

                                        at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:420)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)

                                        at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)

                                        at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:392)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1128)

                                        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2871)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1463)

                                        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:485)

                                        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:773)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getJpsTree(JpsDBDataManager.java:3322)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.checkForEntryExists(JpsDBDataManager.java:3348)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getObjectCountInDoAs(JpsDBDataManager.java:4521)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getObjectCount(JpsDBDataManager.java:4510)

                                        at oracle.security.jps.internal.policystore.entitymanager.impl.InternalQueryManagerImpl.getObjectCount(InternalQueryManagerImpl.java:1383)

                                        at oracle.security.jps.az.internal.management.pd.service.GenericPolicyDistributionService.internalGetApplicationPolicyChangeLog(GenericPolicyDistributionService.java:1520)

                                        at oracle.security.jps.az.internal.management.pd.service.GenericPolicyDistributionService.getApplicationPolicyChangeLog(GenericPolicyDistributionService.java:1484)

                                        at oracle.security.jps.az.internal.management.pd.service.PolicyDistributionServiceImpl.getApplicationPolicyChangeLog(PolicyDistributionServiceImpl.java:258)

                                        at oracle.security.jps.az.internal.runtime.notifier.PolicyChangeNotifierImpl.isSystemChanged(PolicyChangeNotifierImpl.java:175)

                                        at oracle.security.jps.az.internal.runtime.notifier.PolicyChangeNotifierImpl.isChanged(PolicyChangeNotifierImpl.java:154)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScanner.scanSystemPolicy(PolicyChangeScanner.java:260)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScanner.scan(PolicyChangeScanner.java:218)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScannerThread.run(PolicyChangeScannerThread.java:93)

                                at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:811)

                                        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)

                                        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)

                                        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:238)

                                        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)

                                        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)

                                        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)

                                        at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:239)

                                        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)

                                        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)

                                        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:451)

                                        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:343)

                                        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)

                                        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)

                                        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)

                                        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)

                                        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:454)

                                        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:538)

                                        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)

                                        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)

                                        at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123)

                                        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:293)

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1508)

                                        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:579)

                                        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1717)

                                        at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:264)

                                        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:646)

                                        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2611)

                                        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2570)

                                        at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:420)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)

                                        at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)

                                        at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:392)

                                        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1128)

                                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2871)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)

                                        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1463)

                                        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:485)

                                        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:773)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getJpsTree(JpsDBDataManager.java:3322)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.checkForEntryExists(JpsDBDataManager.java:3348)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getObjectCountInDoAs(JpsDBDataManager.java:4521)

                                        at oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.getObjectCount(JpsDBDataManager.java:4510)

                                        at oracle.security.jps.internal.policystore.entitymanager.impl.InternalQueryManagerImpl.getObjectCount(InternalQueryManagerImpl.java:1383)

                                        at oracle.security.jps.az.internal.management.pd.service.GenericPolicyDistributionService.internalGetApplicationPolicyChangeLog(GenericPolicyDistributionService.java:1520)

                                        at oracle.security.jps.az.internal.management.pd.service.GenericPolicyDistributionService.getApplicationPolicyChangeLog(GenericPolicyDistributionService.java:1484)

                                        at oracle.security.jps.az.internal.management.pd.service.PolicyDistributionServiceImpl.getApplicationPolicyChangeLog(PolicyDistributionServiceImpl.java:258)

                                        at oracle.security.jps.az.internal.runtime.notifier.PolicyChangeNotifierImpl.isSystemChanged(PolicyChangeNotifierImpl.java:175)

                                        at oracle.security.jps.az.internal.runtime.notifier.PolicyChangeNotifierImpl.isChanged(PolicyChangeNotifierImpl.java:154)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScanner.scanSystemPolicy(PolicyChangeScanner.java:260)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScanner.scan(PolicyChangeScanner.java:218)

                                        at oracle.security.jps.az.internal.common.scanner.PolicyChangeScannerThread.run(PolicyChangeScannerThread.java:93)

                                • 13. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                                  TheKop88

                                  Hi ,

                                  Did you reset the password to new one or same old value ?

                                  soa data source file resides in (but password is encrypted here) :->
                                  <DomainHome>/config/jdbc/SOADataSource-jdbc.xml

                                  To start the server, you can Reset the schema password to the old one. The server will get started. If you want to change the schema password later, first modify it in the weblogic administration console and then the actual schema password
                                  Thanks

                                  • 14. Re: The weblogic Admin server starting process stuck at <BEA-002900> <Initializing self-tuning thread pool>
                                    1556169

                                    Hi,

                                     

                                    We have asked to reset the password to same old value.

                                    1 2 Previous Next