Unable to access anything related to OBIS

Adam Wickes

Hi all,

We seem to have an issue with our 12.2.1.4 windows install.
The first indication was that we could no longer log in presentation services.

After checking EM, I noticed that everything all looked fine in there so I decided to restart the server and see if that would help.
Nothing changed.

I then attempted to login to the online rpd using the admin tool and got the following error:

My next thought was that the rpd might be corrupt so I attempted to use datamodel.cmd to upload an rpd that I knew worked.

I then confirmed that the port was indeed open and that everything was running correctly which it was.

I have checked most logs... the most interesting being bi_server1-diagnostic.

[2019-08-08T18:14:40.405+10:00] [bi_server1] [NOTIFICATION] [] [oracle.bi.utilitycomm.SocketPool4$SocketPoolClient] [tid: pool-113-thread-1] [userId: weblogic] [ecid: 05c88e1b-7cb5-450d-8868-d11f732d7370-0000061f,0:107] [APP: bi-lcm-rest] [partition-name: DOMAIN] [tenant-name: GLOBAL] [SI-Key: ssi] maxTry=3,i=2,oracle.bi.utilitycomm.ObjectPoolCreateException: create object failed java.net.ConnectException: Connection refused: connect[[  at oracle.bi.utilitycomm.SocketPool4$SocketPoolClient.create(SocketPool4.java:246)  at oracle.bi.utilitycomm.ObjectPoolFactory.createObjects_locked(ObjectPoolFactory.java:117)  at oracle.bi.utilitycomm.ObjectPoolFactory.getObject(ObjectPoolFactory.java:169)  at oracle.bi.utilitycomm.SocketPool4$SocketPoolClient.getObject(SocketPool4.java:254)  at oracle.bi.utilitycomm.SocketPool4$SocketPoolClient.getFreeRpcClient(SocketPool4.java:302)  at oracle.bi.utilitycomm.SocketPool4.getFreeRpcClient(SocketPool4.java:409)  at oracle.bi.jdbcclient.TcpClientAPI.callRPC(TcpClientAPI.java:55)  at oracle.bi.jdbcclient.TcpClientAPI.OpenServer(TcpClientAPI.java:169)  at oracle.bi.jdbcclient.TcpClientAPI.OpenSession(TcpClientAPI.java:98)  at oracle.bi.jdbc.AnaJdbcConnection.<init>(AnaJdbcConnection.java:120)  at oracle.bi.jdbc.AnaJdbcDriver.connectToNQSNode(AnaJdbcDriver.java:399)  at oracle.bi.jdbc.AnaJdbcDriver.connect(AnaJdbcDriver.java:460)  at java.sql.DriverManager.getConnection(DriverManager.java:664)  at java.sql.DriverManager.getConnection(DriverManager.java:208)  at oracle.bi.jdbc.AnaJdbcDataSourceImpl.getConnection(AnaJdbcDataSourceImpl.java:28)  at oracle.bi.jdbc.AnaJdbcDataSourceImpl.getConnection(AnaJdbcDataSourceImpl.java:38)  at oracle.bi.server.customization.utils.BiProcedureExecutor.getConnection(BiProcedureExecutor.java:437)  at oracle.bi.server.customization.utils.BiProcedureExecutor.uploadRpdProcedure(BiProcedureExecutor.java:463)  at oracle.bi.server.customization.utils.UploadBinRpdTaskCSS.execute(UploadBinRpdTaskCSS.java:265)  at oracle.bi.server.customization.utils.CallableProgress.call(CallableProgress.java:37)  at oracle.bi.server.customization.utils.UploadBinRpdTaskCSS.call(UploadBinRpdTaskCSS.java:33)  at oracle.bi.server.customization.utils.CallableProgress.call(CallableProgress.java:12)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)]][2019-08-08T18:14:40.405+10:00] [bi_server1] [WARNING] [] [oracle.bi] [tid: pool-113-thread-1] [userId: weblogic] [ecid: 05c88e1b-7cb5-450d-8868-d11f732d7370-0000061f,0:107] [APP: bi-lcm-rest] [partition-name: DOMAIN] [tenant-name: GLOBAL] [SI-Key: ssi] java.net.ConnectException: Unable to connect to port 9517 on machine xxxxxx.

The logs also seem to be littered with the below error but not sure if it's relevant given we're not running in a cluster.
We are however using Singleton Data Directory and Global Cache but everything seems to be ok with that. No permission issues etc.

The only thing I can think of that may have caused this is the following:

This morning, I attempted to use "export_service_instance" as I wanted to migrate the catalog to another machine.
It failed due to not knowing what the BI_PRODUCT_HOME was.
I then compared the environment variables on the bad machine vs another machine that I knew worked and noticed that the good machine didn't reference ORACLE_HOME, DOMAIN_HOME, BITOOLS_HOME etc etc
Sooooo... Perhaps stupidly, I took notes of what the env var values were, deleted them and restarted the app.

I then re-tried the export process again and it was around this time where something happened I believe.
Now for the really stupid part.... My first reaction was to restart the server to see if that would fix it... Unfortunately, I didn't save the note specifying the env vars.... *sigh*.

Anyway... that's my full disclosure. Some pretty dumb maneuvers on my part so try not to kick me too hard while i'm down.
I just want this bloody thing fixed.

Appreciate any help that you guys can give me.
Cheers,
Adam

PS Would just like to re-iterate that I have restarted about 20 times now and every time, it comes up perfectly with no issues. In EM, everything is green. Just cannot access the rpd and therefore, OBIPS.

Adam Wickes

Looks like some other stuff is not connecting either....

Gianni Ceresa

Hi,
I always stay away as much as I can from OBIEE installed on Windows for ... (good) reasons

Any chance you maybe have a Windows firewall bothering you or something like that? That's the thing which once annoyed me last time I installed on Windows (6+ years ago ).

Adam Wickes

Have had a quick look at the firewall but couldn't seem to see anything.
I might turn it off and retry.

I may not have mentioned before but this server has been working fine for months up until this morning when I attempted the export. Frustrating.

Is there any way of "re-setting" the rpd back to what it was out of the box? Without being able to use datamodel.cmd that is.

Gianni Ceresa

You could try by redeploying the default BAR (the file must be stored somewhere in your install, search for any *.bar file).

https://gerardnico.com/dat/obiee/admin/12c/bar#from_122141

Beware: this command will probably do more than just resetting the RPD, there are chances it will override and bring back to default other things too (I don't remember anymore the details about flags not working fine and the command importing too much or not enough).

Don't you just have a backup of the server you can rollback to? Would maybe be easier all in all based on the time this is taking you ...

[Deleted User]

Gianni started you off in a good direction. Windows can be a questionable proposition as a server OS.

A full BAR replacement will show you whether vanilla works.

If that fails either revert to an earlier backup or be prepared to dig, dig, dig. Last time I has this issue O simply re-installed. Saved everybody a lot of pain.

[Deleted User]

Gianni started you off in a good direction. Windows can be a questionable proposition as a server OS.

A full BAR replacement will show you whether vanilla works.

If that fails either revert to an earlier backup or be prepared to dig, dig, dig. Last time I has this issue O simply re-installed. Saved everybody a lot of pain.

[Deleted User]

Any progress Adam?

Adam Wickes

Christian Berg wrote:Any progress Adam?

Ok, so yes... Great progress actually. Sorry I haven't been back to reply yet. Was up until all hours with Oracle support last night trying to nut it out.

Went through a lot of changes including changing the obis port number, NQSConfig server thread stack size increase etc etc

I was about to give up when I happened to stumble across an error in the server logs that suggested that nqsserver.exe was consistently crashing.
After mentioning this to support, they were able to find an existing bug raised recently to do with that error, 12.2.1.4 and Windows 2012 R2.

Oracle support response:

This is a known BUG ( Bug 28728503 ) which occurs due to mis-configuration in javads internal file. It usually happens when you import/export the service instance. 

Fix:

1. Stop the OBIEE services 2. Update the obis.properties file (user_projects\domains\bi\config\fmwconfig\bienv\OBIS\obis.properties ) with the following env variable and its value to TRUE : ORACLE_BI_JAVADS_METADATA_FORCEREFRESH_OFF=TRUE 3. Restart the obiee services 4. Confirm from the obis1.out that the environment variable ORACLE_BI_JAVADS_METADATA_FORCEREFRESH_OFF is set to TRUE 5. Test /analytics login

Bug:

https://support.oracle.com/epmos/faces/BugDisplay?id=28728503&parent=SrDetailText&sourceId=3-20719240211

Very happy camper here!!

Adam Wickes

Gianni Ceresa wrote:You could try by redeploying the default BAR (the file must be stored somewhere in your install, search for any *.bar file).https://gerardnico.com/dat/obiee/admin/12c/bar#from_122141 Beware: this command will probably do more than just resetting the RPD, there are chances it will override and bring back to default other things too (I don't remember anymore the details about flags not working fine and the command importing too much or not enough).

You mention flags not working correctly. Has this been an issue in the community?

[Deleted User]

Nice one!!