12 Replies Latest reply: Sep 13, 2013 8:17 AM by Mike Paladino RSS

    Planning services getting down with many users logging in

    904145
      Hi

      We are facing an issue with Hyperion Planning application wherin if more than 12 users are hitting for a data form Planning services are going down. With 11 users it is working fine. I checked the planning logs and i am getting this exception: (We are using 11.1.2.1)

      ####<Dec 7, 2011 12:38:34 PM EST> <Error> <HTTP> <server name> <Planning0> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1323279514131> <BEA-101020> <[ServletContext@1397642062[app:PLANNING module:HyperionPlanning path:/HyperionPlanning spec-version:2.5 version:11.1.2.0]] Servlet failed with Exception
      java.lang.NullPointerException
           at com.hyperion.planning.HyperionPlanningBean.getMaxCellTextSize(Unknown Source)
           at jsp_servlet.__enterdata_inner._jspService(__enterdata_inner.java:978)
           at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
           at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
           at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
           at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
           at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
           at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
           at HspValidationFilter.doFilter(Unknown Source)
           at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
           at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
           at java.security.AccessController.doPrivileged(AccessController.java:284)
           at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
           at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
           at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
           at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
           at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
           at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
           at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
           at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
           at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
           at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
           at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
           at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
           at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
           at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
           at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

      Any suggestions on this will be of great help.

      Thank you
        • 1. Re: Planning services getting down with many users logging in
          Robb Salzmann
          Try increasing the heap (memory) settings on the JVM for your planning web app.

          Regards,
          Robb Salzmann
          • 2. Re: Planning services getting down with many users logging in
            904145
            Hi Robb

            Thanks for the reply.. Where to find this JVM setting .. (11.1.2). I have already increased the heap size as : Xms512m xmx 1512 (more than 1 gb) permsize=64m maxpermsize=256m for Planning. Should this not be enough for 10 -15 users.. for 64 bit machine.. Also do i need to increase this more since i read it in one of the posts that weblogic does not supports more than 1.5 GB.
            Will there be any help if i change the permsize and maxpermsize.

            Thank You

            Sandeep Kumar Upadhyay

            Edited by: 901142 on Dec 8, 2011 9:49 AM
            • 3. Re: Planning services getting down with many users logging in
              Mike Paladino

              I am seeing the same issue, not with user counts but the same error, did memory resolve the issue I am set to 1537M and only have a few users.

              • 4. Re: Planning services getting down with many users logging in
                Sh!va

                Which version you are on?? Is it a new installation ??

                 

                Cheers!

                 

                Sh!va

                • 5. Re: Planning services getting down with many users logging in
                  Mike Paladino

                  11.1.2.1.600

                   

                  Weve had it for about a year, just seeing these recently

                  • 6. Re: Planning services getting down with many users logging in
                    Sh!va

                    Can you share logs as well...Also any changes from when u are seeing such issues ..Any new webform created ??

                     

                     

                    Cheers!

                     

                    Sh!va

                    • 7. Re: Planning services getting down with many users logging in
                      Mike Paladino

                      absolutely, I am on the Infra side of things so I can tell you what I see

                      What drove me to finding this is we had a user call about slowness in a form driven by a task list using a link to a differnet application, the form was opening fine directly but when we opened it through the task list invoking the link we encountered intermittent slowness.  I went out to the logs and noticed a ton of the exceptions below.  I am not sure if it is related as the slowness is intermittent and not always reported, my guess is it is not because of the sheer volume of them but not sure so I figured I mention it

                       

                      ####<Sep 11, 2013 2:51:55 PM EDT> <Notice> <Diagnostics> <servername> <Planning2> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1378925515427> <BEA-320068> <Watch 'UncheckedException' with severity 'Notice' on server 'Planning2' has triggered at Sep 11, 2013 2:51:55 PM EDT. Notification details:

                      WatchRuleType: Log

                      WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))

                      WatchData: DATE = Sep 11, 2013 2:51:55 PM EDT SERVER = Planning2 MESSAGE = [ServletContext@370778047[app:PLANNING module:HyperionPlanning path:/HyperionPlanning spec-version:2.5 version:11.1.2.0]] Servlet failed with Exception

                      java.lang.NullPointerException

                              at jsp_servlet.__tl_navigator._jspService(__tl_navigator.java:1637)

                              at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)

                              at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

                              at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

                              at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)

                              at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)

                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

                              at HspValidationFilter.doFilter(Unknown Source)

                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

                              at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)

                              at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)

                              at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)

                              at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)

                              at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)

                              at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)

                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

                              at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)

                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

                              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)

                              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)

                              at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

                              at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

                              at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)

                              at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)

                              at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)

                              at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)

                              at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

                      SUBSYSTEM = HTTP USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-101020 MACHINE = servername TXID =  CONTEXTID =  TIMESTAMP = 1378925515418

                      WatchAlarmType: AutomaticReset

                      WatchAlarmResetPeriod: 30000

                      >

                      ####<Sep 11, 2013 2:51:56 PM EDT> <Info> <JDBC> <cphypdv3v.sherwin.com> <Planning2> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1378925516620> <BEA-001128> <Connection for pool "EPMSystemRegistry" closed.>

                      ####<Sep 11, 2013 2:51:58 PM EDT> <Alert> <Diagnostics> <cphypdv3v.sherwin.com> <Planning2> <oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Wed Sep 11 14:51:58 EDT 2013)> <<WLS Kernel>> <> <> <1378925518582> <BEA-320016> <Creating diagnostic image in /swpkg/oracle/hyperion/Oracle/Middleware/user_projects/domains/EPMSystem/servers/Planning2/adr/diag/ofm/EPMSystem/Planning2/incident/incdir_1419 with a lockout minute period of 1.>

                      • 8. Re: Planning services getting down with many users logging in
                        Mike Paladino

                        I am going to create a new post on this, I dont think it is related to this issue originally posted

                        • 9. Re: Planning services getting down with many users logging in
                          Sh!va

                          Thats ok ..if you want to open new post...But this is also not resolved yet..

                           

                          I would suggest , to avoid using task list for a day and keep on opening that webform and monitor the JVM...

                           

                          We should also look on optimizing the webform by simple funda's like " Supress missing" , " Fit to default/middium" -  can't  able to recall the exact name and also design of webform

                           

                          Cheers!

                           

                          Sh!va

                          • 10. Re: Planning services getting down with many users logging in
                            Mike Paladino

                            thanx, ill relay that to dev

                            • 11. Re: Planning services getting down with many users logging in
                              AmarnathK

                              was there any Patch update recently? Any recent changes from infra side (server related / planning application related)

                               

                              Amarnath

                              http://amarnath-essbase-blog.blogspot.com

                               


                              • 12. Re: Planning services getting down with many users logging in
                                Mike Paladino

                                Nothing recently, a few months ago we put these in

                                Patch NameDescription
                                16171421Patch 16171421: PATCH SET UPDATE: HYPERION ESSBASE ADMINISTRATION SERVICES SERVER 11.1.2.1.106
                                16171431Patch 16171431: PATCH SET UPDATE: HYPERION ESSBASE SERVER 11.1.2.1.106
                                16171429Patch 16171429: PATCH SET UPDATE: HYPERION ESSBASE RTC 11.1.2.1.106
                                14224664Patch Set Update: 11.1.2.1.600 for Oracle Hyperion Enterprise Performance Management (Patch)
                                14224664Patch Set Update: 11.1.2.1.600 for Oracle Hyperion Enterprise Performance Management (Patch)
                                14832995Patch Set Update: 11.1.2.1.131 for Oracle Hyperion Reporting and Analysis for Web Analysis (Patch)
                                16171393Patch 16171393: PATCH SET UPDATE: HYPERION ANALYTIC PROVIDER SERVICES 11.1.2.1.106
                                14265667Patch Set Update: 11.1.2.1.104 for Oracle Hyperion Calculation Manager (Patch)
                                15993965Patch Set Update: 11.1.2.1.133 for Hyperion Financial Reporting (Patch)
                                13950409Patch Set Update: 11.1.2.1.123 for Oracle Hyperion Reporting and Analysis for Foundation (Patch)
                                14119724Patch Set Update: 11.1.2.1.600 for Oracle Enterprise Performance Management System Workspace (Patch)
                                14464109Patch Set Update: 11.1.2.1.600 Oracle Hyperion Planning, Fusion Edition (Patch)
                                14142678Patch Set Update: 11.1.2.1.600 for Oracle Hypeiron Shared Services (Patch)