7 Replies Latest reply: Oct 3, 2013 8:20 AM by kkwok RSS

    Error starting Foundation Service version 11.1.2.1.600

    Loreto

      Hello,

       

      I'm doing an isntallation of HFM version 11.1.2.1. Everything was working fine, till today.

       

      Yestarday we were implementin SSO and worked fine as well.

       

      But this morning, after restarting all the services, I get an error in foundation log file:

       

      No needs to run custom updater for WORKSPACE#11.1.2.0
      loggingUpdateWORKSPACE.block file exist or the system is running in the Fusion mode, skipping logging.xml configuration
      <10-jul-2013 12H12' CEST> <Error> <HTTP> <BEA-101216> <Servlet: "HubFrameworkServlet" failed to preload on startup in Web application: "interop".
      java.lang.NoSuchMethodError: com/hyperion/css/cache/CacheManager.startDeltaCacheCleanupThread(Ljava/util/Map;)V
      at com.hyperion.hub.struts.action.HUBActionServlet.cleanupDeltaCacheTable(HUBActionServlet.java:395)
      at com.hyperion.hub.struts.action.HUBActionServlet.init(HUBActionServlet.java:102)
      at javax.servlet.GenericServlet.init(GenericServlet.java:241)
      at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      Truncated. see log file for complete stacktrace
      >
      <10-jul-2013 12H12' CEST> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'SHAREDSERVICES [Version=11.1.2.0]'.
      weblogic.application.ModuleException: [HTTP:101216]Servlet: "HubFrameworkServlet" failed to preload on startup in Web application: "interop".
      java.lang.NoSuchMethodError: com/hyperion/css/cache/CacheManager.startDeltaCacheCleanupThread(Ljava/util/Map;)V
      at com.hyperion.hub.struts.action.HUBActionServlet.cleanupDeltaCacheTable(HUBActionServlet.java:395)
      at com.hyperion.hub.struts.action.HUBActionServlet.init(HUBActionServlet.java:102)
      at javax.servlet.GenericServlet.init(GenericServlet.java:241)
      at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
      at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:64)
      at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58)
      at weblogic.servlet.internal.StubLifecycleHelper.<init>(StubLifecycleHelper.java:48)
      at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:539)
      at weblogic.servlet.internal.WebAppServletContext.preloadServlet(WebAppServletContext.java:1985)
      at weblogic.servlet.internal.WebAppServletContext.loadServletsOnStartup(WebAppServletContext.java:1959)
      at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1878)
      at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3153)
      at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1508)
      at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:482)
      at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
      at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
      at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247)
      at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
      at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:27)
      at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:636)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:205)
      at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:58)
      at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
      at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
      at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
      at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
      at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51)
      at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200)
      at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
      at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:261)
      at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:220)
      at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
      at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
      at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
      at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
      at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

      at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1510)
      at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:482)
      at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
      Truncated. see log file for complete stacktrace
      Caused By: java.lang.NoSuchMethodError: com/hyperion/css/cache/CacheManager.startDeltaCacheCleanupThread(Ljava/util/Map;)V
      at com.hyperion.hub.struts.action.HUBActionServlet.cleanupDeltaCacheTable(HUBActionServlet.java:395)
      at com.hyperion.hub.struts.action.HUBActionServlet.init(HUBActionServlet.java:102)
      at javax.servlet.GenericServlet.init(GenericServlet.java:241)
      at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      Truncated. see log file for complete stacktrace
      >
      <10-jul-2013 12H12' CEST> <Alert> <Diagnostics> <BEA-320016> <Creating diagnostic image in e:\hyperion\oracle\middleware\user_projects\domains\epmsystem\servers\foundationservices0\adr\diag\ofm\epmsystem\foundationservices0\incident\incdir_20 with a lockout minute period of 1.>
      <10-jul-2013 12H12' CEST> <Notice> <Cluster> <BEA-000197> <Listening for announcements from cluster using unicast cluster messaging>
      <10-jul-2013 12H12' CEST> <Notice> <Cluster> <BEA-000133> <Waiting to synchronize with other running members of FoundationServices.>
      <10-jul-2013 12H12' CEST> <Warning> <JMX> <BEA-149515> <An error was encountered getting the attribute AdminServerListenPort on the MBean com.bea:Name=FoundationServices0,Type=ServerRuntime during a call to getAttributes>
      <10-jul-2013 12H12' CEST> <Warning> <JMX> <BEA-149515> <An error was encountered getting the attribute AdminServerHost on the MBean com.bea:Name=FoundationServices0,Type=ServerRuntime during a call to getAttributes>
      <10-jul-2013 12H13' CEST> <Warning> <Log Management> <BEA-170011> <The LogBroadcaster on this server failed to broadcast log messages to the admin server. The Admin server may not be running. Message broadcasts to the admin server will be disabled.>
      <10-jul-2013 12H13' CEST> <Error> <Diagnostics> <BEA-320127> <An error occurred while generating Image Source configuration as part of the diagnostic image zip file:
      weblogic.diagnostics.image.ImageSourceCreationException: java.lang.NullPointerException
      at weblogic.management.provider.internal.ConfigImageSource.createDiagnosticImage(ConfigImageSource.java:105)
      at weblogic.diagnostics.image.ImageSourceWork.run(ImageSourceWork.java:111)
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
      Caused By: java.lang.NullPointerException
      at weblogic.management.provider.internal.EditAccessImpl.getEditor(EditAccessImpl.java:597)
      at weblogic.management.provider.internal.ConfigImageSource.writeEditLockState(ConfigImageSource.java:296)
      at weblogic.management.provider.internal.ConfigImageSource.createDiagnosticImage(ConfigImageSource.java:88)
      at weblogic.diagnostics.image.ImageSourceWork.run(ImageSourceWork.java:111)
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
      Truncated. see log file for complete stacktrace

       

      Finally I see RUNNING MODE message.

       

      I can enter in workspace with SSO, open my HFM apps, and reports, and almost everything, excet Shared Services console (error 404)

      Try to redeploy again, with no success

       

      Have you ever seen this error before??

       

      Thank you in advance

       

      REgards

       

      Loreto

        • 1. Re: Error starting Foundation Service version 11.1.2.1.600
          GiacomoL

          Loreto,

          From the error, I'd say your Weblogic classpath got messed up. Interop is looking for a method called "startDeltaCacheCleanupThread" in class com.hyperion.css.cache.CacheManager.


          The DeltaCache is a new feature introduced with 11.1.2.x to manage changes in user provisioning at runtime; it's seen a number of changes between 11.1.2.0 and .3 across various patches, so if you've reused some old JARs while implementing SSO, they might be lacking this feature and being picked up in the classpath ahead of recent ones.

          • 2. Re: Error starting Foundation Service version 11.1.2.1.600
            Loreto

            Hi GiacomoL

             

            thank you for you answer

             

            Well, it is true that I added CustomAuth.jar file in

             

            <MIDDLEWARE_HOME>\EPMSystem11R1\common\jlib\11.1.2.0 in all the three servers

            Should I delete this files  and try again??


            Is there anyway to have weblogic classpath using those new .jar files (implementing the new DeltaCache feature) and configured SSO as well??


            Thank you in advance


            REgards


            Loreto

            • 3. Re: Error starting Foundation Service version 11.1.2.1.600
              GiacomoL

              Well, it depends on what is in that jar -- it's custom-compiled code, after all. I guess it was compiled for an older version.

               

              You could try enabling Custom Authentication on a different directory (I assume you've enabled it for Native at the moment) and restarting; if still it doesn't work, take the jar out and restart -- if that works, the problem really is the jar, and chances are that it needs to be recompiled/repackaged against the new libraries.

              • 4. Re: Error starting Foundation Service version 11.1.2.1.600
                Loreto

                Hi GiacomoL

                 

                Well I try to remove .jar files from all the servers, and restart again Foundation service, but with no success, I have the same error message in the log file

                 

                 

                Do you know if there is something i can do with WebLogic ClassPath??

                 

                Thanks again

                 

                Regars

                 

                Loreto

                • 5. Re: Error starting Foundation Service version 11.1.2.1.600
                  kkwok

                  Hi LdeM,

                   

                  I have the same problem in the logs, did you find a solution to this?

                   

                  Thanks!

                  • 6. Re: Error starting Foundation Service version 11.1.2.1.600
                    Loreto
                    Hi kkwok,
                    Yes we finally found a solution:
                    rollback HSS patches #14142678 (and  #14740171)
                    We recreated CSS_GROUP_CACHE_DELTA table
                    Redeploy Foundation
                    Start agian services and this time was succesfully.

                    We've apply again patches #14142678 (and  #14740171), recreate the CSS_GROUP_CACHE_DELTA table and redeploying as well. And the start was successfully one more time
                    • 7. Re: Error starting Foundation Service version 11.1.2.1.600
                      kkwok

                      Hey LdeM,

                       

                      Thanks for your reply, Oracle support just got us a workaround yesterday as well. But they haven't specify what's the cause of this. they were just comparing the .jar

                       

                      Resolution:

                      1.     Deleted the css.jar from Oracle/Middleware/EPMSystem11R1/common/jlib/11.1.2.0

                      2.     Deleted the tmp directory under Oracle\Middleware\user_projects\domains\epmsystem\servers\FoundationServices0


                      Thanks