Hi,
We'd installed OBIEE 12.2.1.2 on AIX 7.1 about 30 days ago. The BI services (OBIS1, OBISCH1, etc) have crashed 2 times due to insufficient memory. The first crash was on April 4. The second crash was yesterday April 16. In both instances, the admin server and managed server are fine. I could log into Weblogic console and Enterprise Manager. This server has 16 GB of RAM, 8 GB of pagefile, and only 3 users running minimal reports! We adjusted the JVM size according to the 12c tuning guide. (4 GB for admin server, 4 GB for managed server, 2 GB for node manager). Anyone has similar issues or suggestions? We previously had 12.2.1.0 and never encountered this issue.
Sample of the nodemanager.log:
<Apr 16, 2017 9:30:05 PM HST> <INFO> <bi> <obips1> <The server 'obips1' with process id 6684736 is no longer alive; waiting for the process to die.>
<Apr 16, 2017 9:30:05 PM HST> <INFO> <bi> <obips1> <Server failed so attempting to restart (restart count = 1)>
<Apr 16, 2017 9:30:05 PM HST> <SEVERE> <Plugin threw out unexpected exception java.lang.OutOfMemoryError from method oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(Properties props). This is very likely a BUG in plugin implementation and may lead to unexpected behavior. The legal exception allowed from this method is java.io.IOException. The unexpected exception is: java.lang.OutOfMemoryError: fork() failed due to insufficient memory>
java.lang.OutOfMemoryError: fork() failed due to insufficient memory
at weblogic.nodemanager.util.UnixProcessControl.createProcess(UnixProcessControl.java:95)
at weblogic.nodemanager.server.NMProcessNativeImpl.startInternal(NMProcessNativeImpl.java:80)
at weblogic.nodemanager.server.NMProcess.start(NMProcess.java:156)
at oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(WrappedProcess.java:25)
at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:129)
at weblogic.nodemanager.server.ServerMonitor.startProcess(ServerMonitor.java:638)
at weblogic.nodemanager.server.ServerMonitor.runMonitor(ServerMonitor.java:590)
at weblogic.nodemanager.server.ServerMonitor.run(ServerMonitor.java:476)
at java.lang.Thread.run(Thread.java:785)
<Apr 16, 2017 9:30:05 PM HST> <SEVERE> <bi> <obips1> <Unexpected error while monitoring server>
java.io.IOException: Unexpected exception from Plugin: java.lang.OutOfMemoryError: fork() failed due to insufficient memory
at weblogic.nodemanager.server.DecoratedSystemComponentManager.logAndWrapAsIOExceptionIfNecessary(DecoratedSystemComponentManager.java:106)
at weblogic.nodemanager.server.DecoratedSystemComponentManager.access$000(DecoratedSystemComponentManager.java:16)
at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:131)
at weblogic.nodemanager.server.ServerMonitor.startProcess(ServerMonitor.java:638)
at weblogic.nodemanager.server.ServerMonitor.runMonitor(ServerMonitor.java:590)
at weblogic.nodemanager.server.ServerMonitor.run(ServerMonitor.java:476)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.lang.OutOfMemoryError: fork() failed due to insufficient memory
at weblogic.nodemanager.util.UnixProcessControl.createProcess(UnixProcessControl.java:95)
at weblogic.nodemanager.server.NMProcessNativeImpl.startInternal(NMProcessNativeImpl.java:80)
at weblogic.nodemanager.server.NMProcess.start(NMProcess.java:156)
at oracle.bi.cam.obips.plugin.nodemanager.WrappedProcess.start(WrappedProcess.java:25)
at weblogic.nodemanager.server.DecoratedSystemComponentManager$DecoratedProcess.start(DecoratedSystemComponentManager.java:129)
... 4 more