OBIEE 12.2.1.2 Insufficient memory errors

3310714

Hi,

We'd installed OBIEE 12.2.1.2 on AIX 7.1 about 30 days ago.  The BI services (OBIS1, OBISCH1, etc) have crashed 2 times due to insufficient memory.  The first crash was on April 4.  The second crash was yesterday April 16.  In both instances, the admin server and managed server are fine.  I could log into Weblogic console and Enterprise Manager.  This server has 16 GB of RAM, 8 GB of pagefile, and only 3 users running minimal reports!  We adjusted the JVM size according to the 12c tuning guide.  (4 GB for admin server, 4 GB for managed server, 2 GB for node manager).  Anyone has similar issues or suggestions?  We previously had 12.2.1.0 and never encountered this issue.  

Sample of the nodemanager.log:

<Apr 16, 2017 9:30:05 PM HST> <INFO> <bi> <obips1> <The server 'obips1' with process id 6684736 is no longer alive; waiting for the process to die.>

<Apr 16, 2017 9:30:05 PM HST> <INFO> <bi> <obips1> <Server failed so attempting to restart (restart count = 1)>

<Apr 16, 2017 9:30:05 PM HST> <SEVERE> <Plugin threw out unexpected exception java.lang.OutOfMemoryError from method oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(Properties props). This is very likely a BUG in plugin implementation and may lead to unexpected behavior. The legal exception allowed from this method is java.io.IOException. The unexpected exception is: java.lang.OutOfMemoryError: fork() failed due to insufficient memory>

java.lang.OutOfMemoryError: fork() failed due to insufficient memory

  at weblogic.nodemanager.util.UnixProcessControl.createProcess(UnixProcessControl.java:95)

  at weblogic.nodemanager.server.NMProcessNativeImpl.startInternal(NMProcessNativeImpl.java:80)

  at weblogic.nodemanager.server.NMProcess.start(NMProcess.java:156)

  at oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(WrappedProcess.java:25)

  at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:129)

  at weblogic.nodemanager.server.ServerMonitor.startProcess(ServerMonitor.java:638)

  at weblogic.nodemanager.server.ServerMonitor.runMonitor(ServerMonitor.java:590)

  at weblogic.nodemanager.server.ServerMonitor.run(ServerMonitor.java:476)

  at java.lang.Thread.run(Thread.java:785)

<Apr 16, 2017 9:30:05 PM HST> <SEVERE> <bi> <obips1> <Unexpected error while monitoring server>

java.io.IOException: Unexpected exception from Plugin: java.lang.OutOfMemoryError: fork() failed due to insufficient memory

  at weblogic.nodemanager.server.DecoratedSystemComponentManager.logAndWrapAsIOExceptionIfNecessary(DecoratedSystemComponentManager.java:106)

  at weblogic.nodemanager.server.DecoratedSystemComponentManager.access$000(DecoratedSystemComponentManager.java:16)

  at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:131)

  at weblogic.nodemanager.server.ServerMonitor.startProcess(ServerMonitor.java:638)

  at weblogic.nodemanager.server.ServerMonitor.runMonitor(ServerMonitor.java:590)

  at weblogic.nodemanager.server.ServerMonitor.run(ServerMonitor.java:476)

  at java.lang.Thread.run(Thread.java:785)

Caused by: java.lang.OutOfMemoryError: fork() failed due to insufficient memory

  at weblogic.nodemanager.util.UnixProcessControl.createProcess(UnixProcessControl.java:95)

  at weblogic.nodemanager.server.NMProcessNativeImpl.startInternal(NMProcessNativeImpl.java:80)

  at weblogic.nodemanager.server.NMProcess.start(NMProcess.java:156)

  at oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(WrappedProcess.java:25)

  at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:129)

  ... 4 more

[Deleted User]

And this is precisely why that "tuning guide" is basically context-sensitve knob-twisting. Setting GO_FASTER=42; because it says so in the document does not mean that's correct for your system.

Also: You can easily kill a server with 3 people if you're doing loads of ridiculous charting, PDF exports etc since OBIPS (the component which fails) as well as the OBIJH (charting engine) also are dependent on detailed java configurations.

When does your server crash? What is happening at that moment? What activities are being done when it fails?

3310714

We installed our DEV environment on 02/10/2017.  It has 12GB of RAM and the JVM are 2GB for Node Manager, Managed Server, & Admin Server.  I see the insufficient memory errors in the NodeManager.log on 03/17, 04/01-04, 04/16-17.   I am the only user and I don't do much here.   The errors show up all varying times.

We installed our TEST environment on 03/23/2017.  It has 16GB of RAM and the JVM are 2GB for Node Manager, 4GB for Managed Server, & Admin Server.  I see the insufficient memory errors in the NodeManager.log on 04/04-05, 04/16-17.   There are three 3 and only 1 is active.  The errors show up all varying times.

I opened a ticket with Oracle, but haven't heard back.  I wonder if it's another bug.

Haranath Thota

Hi.. we are even seeing similar issue on 12.2.1.2. Can you let us know if you have got any resolution for this issue. Thanks in advance.

3310714

Hi,

Oracle couldn't figure out.  We just scheduled OBIEE to be bounced once a week and it's been fine since.  You may want to try that.  If once a week isn't enough, try twice.

Haranath Thota

ok thanks for the input