OBIEE 11g Server - "Service Temporarily Unavailable" after 30 mins uptime and multiple restarts

chillychin

Im a bit stumped here

So we recently experienced a power interruption to our servers and our server had to be rebooted.

I restarted our OBIEE 11g services, and everything showed up as "Alive" and was running perfectly fine for 30 minutes or so.

After 30 minutes or so, I suddenly hit the error message on my browser when trying to access /analytics

[Service Temporarily Unavailable]

I tried to reboot the server, and the services this time gracefully. Then after a certain amount of time I again hit "service unavailable"

I was monitoring the obiee-start.log and noticed the following error message

FRViewerPermissionexists in the roles = truejava.io.IOException: Cannot run program "powershell": java.io.IOException: error=2, No such file or directoryContinuing ...java.io.IOException: Cannot run program "powershell": java.io.IOException: error=2, No such file or directoryContinuing ...UserRoles are for user BISystemUser:BISystemUserUserRoles are for user BISystemUser are: [Ljava.lang.String;@4ce4e3c2Role: native://DN=cn=CALCMGR:0001,ou=CALCMGR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=CALCMGR:0002,ou=CALCMGR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=BPM:0004,ou=BPM,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0001,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=BPM:0001,ou=BPM,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0005,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0004,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0003,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0002,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLERole: native://DN=cn=FR:0006,ou=FR,ou=Roles,dc=css,dc=hyperion,dc=com?ROLEFRViewerPermissionexists in the roles = true/home/itadm/u01/obiee/user_projects/domains/bifoundation_domain/bin/startWebLogic.sh: line 180:  1285 Killed                  ${JAVA_HOME}/bin/java ${JAVA_V                                                   M} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}/home/itadm/u01/obiee/user_projects/domains/bifoundation_domain/bin/startWebLogic.sh: line 180:  1809 Killed                  ${JAVA_HOME}/bin/java ${JAVA_V                                                   M} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}+ set +x

The last two lines talking about a process killed is what is concerning me.

I am trying to restart my services and take a look at the logs in weblogic console, but as I am digging the services die on me again.

Im not quite sure what is causing this, as sometimes I get about 30 minutes of uptime, other times I get maybe 2 hours but it eventually dies.

Has anyone seen this before?