Oracle Analytics Cloud and Server

Home› Oracle Analytics› Oracle Analytics Forums› Oracle Analytics Cloud and Server› Oracle Analytics Cloud and Server

Categories

Welcome to the Oracle Analytics Community: Please complete your User Profile and upload your Profile Picture

OAS_6.4: bi_server1 stuck forever in STARTING state.

Accepted answer
451
Views
9
Comments
User_3GIDX
User_3GIDX Rank 1 - Community Starter
edited Feb 2, 2024 1:07PM in Oracle Analytics Cloud and Server

Hi,

We are facing a strange in OAS 6.4 that is bi_server1 stuck forever in STARTING state.

We tried various options with no luck.

Any further inputs would help and much appreciated.

It was working. After a overnight scheduled restart, it never came back in RUNNING state which is strange. No specific changes are made. Connections to RCU also look good.

Checked the logs bi_server1,bi_server1_dignostic logs etc. Found certain errors, however not that useful:

N####<Feb 2, 2024 3:02:41,946 PM CET> <Error> <javax.enterprise.resource.webcontainer.jsf.application> <bi_server1> <[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <19a18df1-29c3-4e94-a2ec-f1818d 0016be-0000009d> <1706882561946> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <Error Loading Facelet cache: oracle.adfinternal.view.faces.facelets.rich.MDSFaceletCache

[2024-02-02T12:24:19.704+01:00] [bi_server1] [ERROR] [] [oracle.biee.search] [tid: BISearchPushCrawler-thread-7] [userId: <anonymous>] [ecid: e086c1d5-fd79-47b5-b4e7-2d44c6b9464c-0000000b,0:782:19:46] [APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed to create system session due to exception: java.net.ConnectException: Connection refused (Connection refused)[[ javax.xml.ws.ProtocolException: java.net.ConnectException: Connection refused (Connection refused) at com.siebel.analytics.web.jaxwsutils.HttpTcpBridge.sendHttpRequest(HttpTcpBridge.java:105)

The log is not moving forward this specific log entry:

####<Feb 2, 2024 4:22:16,965 PM CET> <Info> <WorkManager> <test> <bi_server1> <Timer-2> <<WLS Kernel>> <> <19a18df1-29c3-4e94-a2ec-f1818d0016be-00000002> <1706887336965> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 4 running threads, 3 idle threads, and 61 standby threads>

Thanks and Regards,

Anant.

Best Answer

Answers

  • Gianni Ceresa
    Gianni Ceresa Mod

    Hi Anant,

    You will need to provide a lot more info than this...

    Everything is logged in OAS, if your managed server doesn't start, look at the logs of the managed server, then also have a look at the admin server log to see if there is something going back there, and also have a look at the node manager logs.

  • SteveF-Oracle
    SteveF-Oracle Mod
    edited Feb 2, 2024 1:24PM

    I agree, we would need to see the bi_server1 .out logs, and possibly nodemanager.

    Did it ever start? First time? Something change? Connection to your RCU database good?


    Clear all the logs, here is a quick script (replace [DOMAIN_NAME] with our domain path )

    If windows, replace 'rm' with DEL, and reverse the slashes.

    # not meant to be elegant, quick and dirty, and gets the job done.
#. $HOME/oas.env
#Node Manager
rm [DOMAIN_HOME]/bi/nodemanager/nodemanager.log
#Admin
rm [DOMAIN_HOME]/bi/servers/AdminServer/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/AdminServer/logs/bipublisher/*.log*
rm [DOMAIN_HOME]/bi/servers/AdminServer/logs/*out*

#Managed
rm [DOMAIN_HOME]/bi/servers/bi_server1/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/bi_server1/logs/*.out*
rm [DOMAIN_HOME]/bi/servers/bi_server1/logs/bipublisher/*.log*

#obis - not UTData
rm [DOMAIN_HOME]/bi/servers/obis1/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/obis1/logs/*.out*
rm [DOMAIN_HOME]/bi/servers/obis1/logs/*.json*

#obips
rm [DOMAIN_HOME]/bi/servers/obips1/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/obips1/logs/*.out*

#obiccs
rm [DOMAIN_HOME]/bi/servers/obiccs1/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/obiccs1/logs/*.out*

#obijh
rm [DOMAIN_HOME]/bi/servers/obijh1/logs/*.log*
rm [DOMAIN_HOME]/bi/servers/obijh1/logs/*.out*

#obisch
rm [DOMAIN_HOME]/bi/servers/obisch1/logs/*.log*
rm [DOMAINHOME]/bi/servers/obisch1/logs/*.out

    execute: start.sh

    Check the bi_server1.out logs, if nothing obvious, the collect a diagnostic dump, and log a service request for additional assistance.

    https://docs.oracle.com/en/middleware/bi/analytics-server/administer-oas/collect-diagnostic-bundles.html#GUID-5B5A6544-AC87-442B-B1C2-57782739DC56

  • User_3GIDX
    User_3GIDX Rank 1 - Community Starter

    Hi Steve,

    Thanks for the quick response. Answers to your questions are as mentioned below:

    1. Did it ever start? First time? Something change? Connection to your RCU database good?
    2. Ans->Yes, it was working. We have a scheduled overnight restart post which it never came back, which is strange. No specific changes were made. Will check for the RCU DB connection.
    3. We have checked the logs(bi_server1, bi server dignostic logs etc.) however, it's not moving forward from STATE_PREPARED to STATE_ADMIN state. Ideally after STATE_ADMIN it should move in RUNNING/ACTIVE state.
    4. Error is as mentioned below:

    javax.xml.ws.ProtocolException: java.net.ConnectException: Connection refused (Connection refused)at com.siebel.analytics.web.jaxwsutils.HttpTcpBridge.sendHttpRequest(HttpTcpBridge.java:105)[2024-02-02T12:24:19.704+01:00] [bi_server1] [ERROR] [] [oracle.biee.search] [tid: BISearchPushCrawler-thread-7] [userId: <anonymous>] [ecid: e086c1d5-fd79-47b5-b4e7-2d44c6b9464c-0000000b,0:782:19:46] [APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed to create system session due to exception: java.net.ConnectException: Connection refused (Connection refused)

    Not moving further from the below:

    ####<Feb 2, 2024 4:22:16,965 PM CET> <Info> <WorkManager> <test> <bi_server1> <Timer-2> <<WLS Kernel>> <> <19a18df1-29c3-4e94-a2ec-f1818d0016be-00000002> <1706887336965> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 4 running threads, 3 idle threads, and 61 standby threads>



    Thanks and Regards,

    Anant.

  • Shantaram-Oracle
    Shantaram-Oracle Rank 4 - Community Specialist

    Also the full error stack from the bi_server1-diagnostic.log should point to the cause of the error as well.. something like..

     Caused By: java.net.ConnectException: Tried all: '1' addresses, but could not 
connect over HTTP to server: '<hostname>, port: 

 failed reasons:
  [0] 
 address:'<hostname>/<IP>',port:'xxxx' : 
java.net.ConnectException: Connection refused

    If it continues to be an issue, Please log a service request with MOS for an in-depth investigation into the issue.

  • User_3GIDX
    User_3GIDX Rank 1 - Community Starter

    Hi Steve,

    Thanks for the response.

    Did it ever start? First time? Something change? Connection to your RCU database good?

    ->It was working. After a overnight scheduled restart, it never came back in RUNNING state which is strange. No specific changes are made. Connections to RCU also look good.

    ->Checked the logs bi_server1,bi_server1_dignostic logs etc. Found certain errors, however not that useful:

    N####<Feb 2, 2024 3:02:41,946 PM CET> <Error> <javax.enterprise.resource.webcontainer.jsf.application> <bi_server1> <[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <19a18df1-29c3-4e94-a2ec-f1818d 0016be-0000009d> <1706882561946> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <Error Loading Facelet cache: oracle.adfinternal.view.faces.facelets.rich.MDSFaceletCache

    [2024-02-02T12:24:19.704+01:00] [bi_server1] [ERROR] [] [oracle.biee.search] [tid: BISearchPushCrawler-thread-7] [userId: <anonymous>] [ecid: e086c1d5-fd79-47b5-b4e7-2d44c6b9464c-0000000b,0:782:19:46] [APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed to create system session due to exception: java.net.ConnectException: Connection refused (Connection refused)[[ javax.xml.ws.ProtocolException: java.net.ConnectException: Connection refused (Connection refused) at com.siebel.analytics.web.jaxwsutils.HttpTcpBridge.sendHttpRequest(HttpTcpBridge.java:105)

    The log is not moving forward this specific log entry:

    ####<Feb 2, 2024 4:22:16,965 PM CET> <Info> <WorkManager> <test> <bi_server1> <Timer-2> <<WLS Kernel>> <> <19a18df1-29c3-4e94-a2ec-f1818d0016be-00000002> <1706887336965> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 4 running threads, 3 idle threads, and 61 standby threads>

    State Transition not moving from STATE_PREPARED to STATE_ADMIN

    Thanks and Regards,

    Anant.

  • User_3GIDX
    User_3GIDX Rank 1 - Community Starter

    Hi Steve,

    Thanks for the response.

    Did it ever start? First time? Something change? Connection to your RCU database good?

    ->It was working. After a overnight scheduled restart, it never came back in RUNNING state which is strange. No specific changes are made. Connections to RCU also look good.

    ->Checked the logs bi_server1,bi_server1_dignostic logs etc. Found certain errors, however not that useful:

    N####<Feb 2, 2024 3:02:41,946 PM CET> <Error> <javax.enterprise.resource.webcontainer.jsf.application> <bi_server1> <[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <19a18df1-29c3-4e94-a2ec-f1818d 0016be-0000009d> <1706882561946> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <Error Loading Facelet cache: oracle.adfinternal.view.faces.facelets.rich.MDSFaceletCache

    [2024-02-02T12:24:19.704+01:00] [bi_server1] [ERROR] [] [oracle.biee.search] [tid: BISearchPushCrawler-thread-7] [userId: <anonymous>] [ecid: e086c1d5-fd79-47b5-b4e7-2d44c6b9464c-0000000b,0:782:19:46] [APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed to create system session due to exception: java.net.ConnectException: Connection refused (Connection refused)[[ javax.xml.ws.ProtocolException: java.net.ConnectException: Connection refused (Connection refused) at com.siebel.analytics.web.jaxwsutils.HttpTcpBridge.sendHttpRequest(HttpTcpBridge.java:105)

    The log is not moving forward this specific log entry:

    ####<Feb 2, 2024 4:22:16,965 PM CET> <Info> <WorkManager> <test> <bi_server1> <Timer-2> <<WLS Kernel>> <> <19a18df1-29c3-4e94-a2ec-f1818d0016be-00000002> <1706887336965> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 4 running threads, 3 idle threads, and 61 standby threads>

    State Transition not moving from STATE_PREPARED to STATE_ADMIN

    Thanks and Regards,

    Anant.

  • User_3GIDX
    User_3GIDX Rank 1 - Community Starter

    Hi Steve,

    Thanks for the response.

    Did it ever start? First time? Something change? Connection to your RCU database good?

    ->It was working. After a overnight scheduled restart, it never came back in RUNNING state which is strange. No specific changes are made. Connections to RCU also look good.

    ->Checked the logs bi_server1,bi_server1_dignostic logs etc. Found certain errors, however not that useful:

    N####<Feb 2, 2024 3:02:41,946 PM CET> <Error> <javax.enterprise.resource.webcontainer.jsf.application> <bi_server1> <[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <19a18df1-29c3-4e94-a2ec-f1818d 0016be-0000009d> <1706882561946> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <Error Loading Facelet cache: oracle.adfinternal.view.faces.facelets.rich.MDSFaceletCache

    [2024-02-02T12:24:19.704+01:00] [bi_server1] [ERROR] [] [oracle.biee.search] [tid: BISearchPushCrawler-thread-7] [userId: <anonymous>] [ecid: e086c1d5-fd79-47b5-b4e7-2d44c6b9464c-0000000b,0:782:19:46] [APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed to create system session due to exception: java.net.ConnectException: Connection refused (Connection refused)[[ javax.xml.ws.ProtocolException: java.net.ConnectException: Connection refused (Connection refused) at com.siebel.analytics.web.jaxwsutils.HttpTcpBridge.sendHttpRequest(HttpTcpBridge.java:105)

    The log is not moving forward this specific log entry:

    ####<Feb 2, 2024 4:22:16,965 PM CET> <Info> <WorkManager> <test> <bi_server1> <Timer-2> <<WLS Kernel>> <> <19a18df1-29c3-4e94-a2ec-f1818d0016be-00000002> <1706887336965> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-002959> <Self-tuning thread pool contains 4 running threads, 3 idle threads, and 61 standby threads>

    State Transition not moving from STATE_PREPARED to STATE_ADMIN

    Thanks and Regards,

    Anant.

  • Hui Wang-Oracle
    Hui Wang-Oracle Rank 3 - Community Apprentice

    How is your issue? what's your backend platform? you can check whether any port conflict or also try to backup your tmp folder and then remove it and then verify your start up also