OBIEE 12c goes on strike

Mark.Thompson

This isn't a question, but a description of something that happened to us today with OBIEE 12c.  During class, our OBIEE system stopped responding - cold.  I tried to check the status of the services, but the process said that the NodeManager was not running.  Upon rebooting the server and attempting to start services, the NodeManager started, but the AdminServer insisted on asking for credentials.  I set the boot.properties file to clear-text credentials and tried again, but no luck. 

We fired up a copy of the backup machine, and it exhibited the same problem.  It hadn't even been turned on for two weeks.  Another copy - same problem.  A copy of a similar machine - same problem.  On one of them I was able to get the AdminServer started, but bi_server1 just hung in STARTING status for over an hour.

Nothing I tried work.  But six hours after the trouble began on machine A, machine C suddenly started working properly again.  I wasn't touching it, not even looking at it.  The startup process for BI_Server1, which had been in STARTING status for at least 45 minutes, finally completed, all 5 OBIEE services started properly, and that machine was working.  Then, after I got back to the hotel after dinner, the other two machines that we had spun up earlier in the day were also working.  The only one that absolutely refuses to behave is the original one that we were training with this morning. 

I wish I could report some magic "ah-ha" moment to you, but I cannot.  I don't know what caused it to fail on multiple machines, nor what caused it to resume working on 3 of 4 machines after going on strike for 6 hours. I figured I'd document it here just in case it happens to someone else.  Working with OBIEE 12c first release on Windows Server 2012.