11 Replies Latest reply on Nov 29, 2018 2:36 PM by Raj453

    OBIEE 12c start up error

    Raj453

      Hi Gurus,

       

      we have an issue when trying to start/stop the OBIEE instance. Below is the message upon attempting to start.  Please share inputs. Thanks.

       

      ====================================================================================================================

      [oracle@****** bin]$ ./stop.sh

      Stopping domain; Using domainHome: /u01/app/oracle/product/12.2.1/Middleware_Home/user_projects/domains/bi ...

       

       

      Initializing WebLogic Scripting Tool (WLST) ...

       

       

      Welcome to WebLogic Server Administration Scripting Shell

       

       

      Type help() for help on available commands

       

       

      <Nov 28, 2018 12:26:17 PM EST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>

      <Nov 28, 2018 12:26:17 PM EST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ********* to **********. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>

      <Nov 28, 2018 12:26:17 PM EST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$NullHostnameVerifier.>

      Reading domain...

      /Servers/AdminServer/ListenPort=9500

      Accessing admin server using URL t3://********:9500

      Start Admin Server connect Exception caught Error occurred while performing connect : Error getting the initial context. There is no server running at t3://*********:9500 : Failed to initialize JNDI context, tried 2 time or times totally, the interval of each time is 0ms.

      t3://*********:9500: Destination ************, 9500 unreachable.; nested exception is:

              java.net.ConnectException: Connection refused (Connection refused); No available router to destination.; nested exception is:

              java.rmi.ConnectException: No available router to destination.

      Use dumpStack() to view the full stacktrace :

      Reading domain...

      /Servers/AdminServer/ListenPort=9500

      Accessing admin server using URL t3://*********:9500

      Requesting credentials ...

      Enter Weblogic login details at prompt

      Weblogic Username: weblogic

      Weblogic Password:

      Starting AdminServer ...

      Unable to connect to AdminServer to shutdown all servers

      Problem invoking WLST - Traceback (innermost last):

        File "/u01/app/oracle/product/12.2.1/Middleware_Home/bi/modules/oracle.bi.sysman/scripts/stop_servers.py", line 188, in ?

        File "/u01/app/oracle/product/12.2.1/Middleware_Home/bi/modules/oracle.bi.sysman/scripts/stop_servers.py", line 80, in stopManagedServersAndSystemComponents

      AttributeError: 'NoneType' object has no attribute 'getAdminServerName'

       

       

      Stopping Derby with pid: 14962; killing (SIGTERM) ...

      [oracle@************* bin]$

        • 1. Re: OBIEE 12c start up error
          Raj453

          Hi.  any suggestions please ..

          • 2. Re: OBIEE 12c start up error
            Charles M

            Could you check the admin server log(s)?

             

            Also, see if this note helps you: weblogic.Admin PING fails with java.net.SocketException: Connection reset; No available router to destination (Doc ID 1398139.1)

             

            Regards,

             

            Charles

            • 3. Re: OBIEE 12c start up error
              Charles M

              Also ... where is that output coming from? Is that standard out, or from a log? I'm a bit confused because you show a stop command, but I can see entries trying to start the admin server? Can you confirm that is the exact output you get, once you execute the stop command.

               

              Also, also, explain more about your configuration. SSL-enabled?, etc.

               

              Better explanations/better answers

              • 4. Re: OBIEE 12c start up error
                Raj453

                Hi Charles,

                 

                the URL you shared prompting for support identifier, i am not using corporate account,  can you please share the PDF or Word version of that link. Thanks.

                • 5. Re: OBIEE 12c start up error
                  Charles M

                  Sorry, can't do that. That is a My Oracle Support note. This is a public forum.

                   

                  My recommendation is that you provide some additional details, so others can help out. Give some context ... when did this last work, any changes made, OS your using, etc. Is this a fresh install (you get the idea)?

                  • 6. Re: OBIEE 12c start up error
                    Raj453

                    Hi Charles,

                     

                    this is on Oracle linux  (on a VM),  not a new install, previously working fine,

                     

                    tried some suggestions like  kill all the oracle user processes,  deleted Admin server  "lok"  files (only after confirming admin server not running using netstat ),  replaced system generated weblogic username and password in "boot.properties" file with the actual values.  Still no luck, 

                    • 7. Re: OBIEE 12c start up error
                      Raj453

                      Hi Charles,

                       

                      From Admin server log file, copying some of the error messages , please take a look.

                       

                      <Nov 28, 2018 4:55:54,825 PM EST> <Error> <Diagnostics> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <8f6393a9-8666-4129-8edf-ba3b13f230f4-00000001> <1543442154825> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320185> <Failed to open diagnostic store file /u01/app/oracle/product/12.2.1/Middleware_Home/user_projects/domains/bi/servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS.DAT after 10 attempts. Some diagnostics features will be unavailable.

                      weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.

                       

                      <Nov 28, 2018 4:56:57,278 PM EST> <Warning> <JMX> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <8f6393a9-8666-4129-8edf-ba3b13f230f4-00000006> <1543442217278> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://172.29.16.181:9500/jndi/weblogic.management.mbeanservers.runtime.>

                      ####<Nov 28, 2018 4:56:57,295 PM EST> <Critical> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <8f6393a9-8666-4129-8edf-ba3b13f230f4-00000007> <1543442217295> <[severity-value: 4] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000386> <Server subsystem failed. Reason: A MultiException has 22 exceptions.  They are:

                      1. java.lang.NullPointerException

                       

                      <Nov 28, 2018 4:56:57,312 PM EST> <Error> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <8f6393a9-8666-4129-8edf-ba3b13f230f4-00000007> <1543442217312> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000383> <A critical service failed. The server will shut itself down.>

                       

                      <Nov 28, 2018 4:58:28,254 PM EST> <Error> <Diagnostics> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a070f2d1-faf8-4eab-a331-cfdc9a2d4181-00000001> <1543442308254> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320185> <Failed to open diagnostic store file /u01/app/oracle/product/12.2.1/Middleware_Home/user_projects/domains/bi/servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS.DAT after 10 attempts. Some diagnostics features will be unavailable.

                      weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT

                       

                      <Nov 28, 2018 4:59:54,532 PM EST> <Warning> <JMX> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a070f2d1-faf8-4eab-a331-cfdc9a2d4181-00000006> <1543442394532> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://172.29.16.181:9500/jndi/weblogic.management.mbeanservers.runtime.>

                      ####<Nov 28, 2018 4:59:54,553 PM EST> <Critical> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <a070f2d1-faf8-4eab-a331-cfdc9a2d4181-00000007> <1543442394553> <[severity-value: 4] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000386> <Server subsystem failed. Reason: A MultiException has 22 exceptions.  They are:

                       

                      <Nov 28, 2018 5:01:34,207 PM EST> <Warning> <JMX> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2a692f87-4fd5-42d5-9b96-f8a098db5a8f-00000005> <1543442494207> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://172.29.16.181:9500/jndi/weblogic.management.mbeanservers.runtime.>

                      ####<Nov 28, 2018 5:01:34,244 PM EST> <Critical> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <2a692f87-4fd5-42d5-9b96-f8a098db5a8f-00000006> <1543442494244> <[severity-value: 4] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000386> <Server subsystem failed. Reason: A MultiException has 6 exceptions.  They are:

                       

                      <Nov 28, 2018 5:03:34,428 PM EST> <Error> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <2515cf00-51f2-4e19-bc67-e5e3ffcfa333-00000007> <1543442614428> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000383> <A critical service failed. The server will shut itself down.>

                      ####<Nov 28, 2018 5:03:34,431 PM EST> <Notice> <WebLogicServer> <Server or Host name> <AdminServer> <main> <<WLS Kernel>> <> <2515cf00-51f2-4e19-bc67-e5e3ffcfa333-00000007> <1543442614431> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>

                      ####<Nov 28, 2018 5:03:34,431 PM EST> <Error> <Kernel> <Server or Host name> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2515cf00-51f2-4e19-bc67-e5e3ffcfa333-00000008> <1543442614431> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000802> <ExecuteRequest failed

                      A MultiException has 1 exceptions

                      • 8. Re: OBIEE 12c start up error
                        Gianni Ceresa

                        Hi,

                        Raj453 wrote:

                         

                        Hi Gurus,

                         

                        we have an issue when trying to start/stop the OBIEE instance. Below is the message upon attempting to start. Please share inputs. Thanks.

                         

                        ====================================================================================================================

                        [oracle@****** bin]$ ./stop.sh

                        Stopping domain; Using domainHome: /u01/app/oracle/product/12.2.1/Middleware_Home/user_projects/domains/bi ...

                         

                         

                        I just had a quick read through the rest of the thread as I blocked on that: you call stop.sh to attempt to start OBIEE? Very little chances it will work and of course the output is lot of issues because nothing is running ...

                         

                        So: what is the output when you call start.sh ?

                        • 9. Re: OBIEE 12c start up error
                          Raj453

                          Hi Gianni,

                           

                          My mistake, i mean for stopping i ran stop.sh.

                           

                          When i attempt to start,  please see below

                           

                          [oracle@Server or Host name bin]$ ./start.sh

                          Starting domain; Using domainHome: /u01/app/oracle/product/12.2.1/Middleware_Home/user_projects/domains/bi ...

                           

                           

                          Initializing WebLogic Scripting Tool (WLST) ...

                           

                           

                          Welcome to WebLogic Server Administration Scripting Shell

                           

                           

                          Type help() for help on available commands

                           

                           

                          <Nov 29, 2018 8:40:53 AM EST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check                                                                    , specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>

                          <Nov 29, 2018 8:40:53 AM EST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this chan                                                                    ge, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>

                          <Nov 29, 2018 8:40:53 AM EST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVeri                                                                    fier$NullHostnameVerifier.>

                          NodeManager already running

                          Reading domain...

                          /Servers/AdminServer/ListenPort=9500

                          Accessing admin server using URL t3://Server or Host name:9500

                          Starting AdminServer ...

                          ^[[B^[[BAdmin Server Start Exception: Error occurred while performing nmStart : Error Starting server AdminServer : Received error message from Node Manager Server: [Server start command for WebLogic server 'AdminServer' failed due to: [Server failed to start up but Node Manager was not aware of the reason]. Please check Node Manager log and/or server 'AdminServer' log for detailed information.]. Please check Node Manager log for details.

                          Use dumpStack() to view the full stacktrace :

                          Reading domain...

                          /Servers/AdminServer/ListenPort=9500

                          Accessing admin server using URL t3://Server or Host name:9500

                          Requesting credentials ...

                          Enter Weblogic login details at prompt

                          Weblogic Username: weblogic

                          Weblogic Password:

                          Starting AdminServer ...

                          Unable to connect to AdminServer on host: Server or Host name

                           

                           

                          Failed to start one or more Servers

                           

                           

                          /Servers/AdminServer/ListenPort=9500

                          Accessing admin server using URL t3://Server or Host name:9500

                          Start Admin Server connect Exception caught Error occurred while performing connect : Error getting the initial context. There is no server running at t3://Server or Host name:9500 : Failed to initialize JNDI context, tried 2 time or times totally, the interval of each time is 0ms.

                          t3://Server or Host name:9500: Destination 172.29.16.181, 9500 unreachable.; nested exception is:

                                  java.net.ConnectException: Connection refused (Connection refused); No available router to destination.; nested exception is:

                                  java.rmi.ConnectException: No available router to destination.

                          Use dumpStack() to view the full stacktrace :

                          Reading domain...

                          Error: runCmd() failed. Do dumpStack() to see details.

                          Failed to get Status of Servers and System Components

                          [oracle@Server or Host name bin]$

                          • 10. Re: OBIEE 12c start up error
                            Charles M

                            Hi,

                             

                            Can you take a look at the node manager log, and paste any errors?

                             

                            Nodemanager: [DOMAIN_HOME]/nodemanager/nodemanager.log

                             

                            Regards,

                             

                            Charles

                            • 11. Re: OBIEE 12c start up error
                              Raj453

                              Hi Charles,

                               

                              dont see any errors in nodemanager log,  only one warning as below

                               

                              <WARNING> <Server start command for WebLogic server 'AdminServer' failed due to: [Server failed to start up but Node Manager was not aware of the reason]. Please check Node Manager log and/or server 'AdminServer' log for detailed information.>

                               

                              Thanks