3 Replies Latest reply: Oct 4, 2013 2:50 PM by $anty RSS

    Web Analysis server crashing EPM 11.1.1.3

    Dave Long

      We have been experiencing issues with Web Analysis crashing on the server side. When this happens, there are no clear indicators why it has gone down. Logs are relatively bare. There's nothing we know of being indicated to users, but the task goes away and we have to restart it. We DO get a core dump file in the WebLogic9 directory path. With approximately 100 users, we're seeing it happen about three times a week. We've given it lots of Java memory (2048MB for both Xms and Xmx) to use thinking that was the issue but it still hasn't corrected it. As of now, we're running a task to restart the server task when it goes missing. Suboptimal solution, obviously. Oracle support hasn't been helpful.

       

      We've got a plan to upgrade within the next six months, but holidays are coming fast. Would like to fix it if possible.

       

      We're running EPM 11 on a PS702, AIX 6.1 with 32GB of RAM. Anyone encounter anything like this? Thanks!

       

      --Dave

        • 1. Re: Web Analysis server crashing EPM 11.1.1.3
          $anty

          So the web analysis crashes even if users are Idle? 
          If yes, this issue could be related to Garbage Collection(GC). GC is not working properly to make the free space for objects in JVM used by Web analysis.

          Also, can you post what exact error message that you are seeing in Webanalysis or Weblogic logs?  Always take backup of logs before restarting services.

           

          Regards,

          Santy

          • 2. Re: Web Analysis server crashing EPM 11.1.1.3
            Dave Long

            Today when it crashed in the morning we had just restarted it about three hours before. There were six users active on the Essbase database (we only access Essbase via Web Analysis Studio on the user side so it's one to one depending on the number of DB connections for that user). It should have been very low volume and I'd be surprised we ate that much memory that fast, but maybe that's where I don't understand what's happening in Oracle's application that can send us to the max and blow it up?

             

            As for logging, there's been no indication in the WebLogic WebAnalysis application log before the problem occurs. Here's the last half hour or so before the crash this morning...

             

            ####<Oct 4, 2013 7:57:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            87827457> <BEA-310002> <13% of the total memory in the server is free>

            ####<Oct 4, 2013 8:19:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            89147529> <BEA-310002> <92% of the total memory in the server is free>

            ####<Oct 4, 2013 8:28:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            89687558> <BEA-310002> <75% of the total memory in the server is free>

            ####<Oct 4, 2013 8:35:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            90107589> <BEA-310002> <64% of the total memory in the server is free>

            ####<Oct 4, 2013 8:39:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            90347600> <BEA-310002> <49% of the total memory in the server is free>

            ####<Oct 4, 2013 8:47:07 AM EDT> <Info> <Health> <xxxxxxx> <WebAnalysis> <weblogic.GCMonitor> <<anonymous>> <> <> <13808

            90827630> <BEA-310002> <32% of the total memory in the server is free>

             

            xxxxxxx was our server name that I retracted from the log messages.

             

            I appreciate the reply!

            • 3. Re: Web Analysis server crashing EPM 11.1.1.3
              $anty

              Above entries are not giving any clue. The logs should have an <Error> entry which caused the Webanalysis managed server to crash.

              Check logs as soon as you see Webanalysis has crashed, backup logs & check them.

              See if you find anything in AdminServer.log

              Also, are there any specific huge reports you think that might cause this? Because that could be also 1 reason which is causing service crash.

               

              I suspect this is outofmemory issue. Consult oracle support with optimal values for Xms & Xmx.  I think 2048 MB is much higher & not recommended.

              Try Xms as 512MB & Xmx as 1536MB. 

               

              Regards,

              Santy.