7 Replies Latest reply: Nov 21, 2012 10:35 AM by 974378 RSS

    ESSSVR.exe crashes after service start

    974378
      Greeting everyone,

      We have a dedicated Essbase server. Today we started getting a TCP/IP protocol timeout error when trying to connect to EAS.
      The first thing we noticed was the "OPMN service for Essbase" was actually running --this is the only Essbase service on this server.
      At first we tried to restart the service, but the problem persisted. Then we tried to restart the server, but this didn't help neither.
      Finally, we noticed that when you launch the service start (by using the oracle provided .bat files), while the service is starting, an ESSSVR.exe process shows up in the task manager, but once the service is started, the ESSSVER.exe process crashes and disappears from the task manager.

      What can be wrong?

      Thank you.

      Regards,

      G.Sanchez
        • 1. Re: ESSSVR.exe crashes after service start
          JohnGoodwin
          Have you checked any of the logs e.g essbase.log and SharedServices_Security_Client.log

          Cheers

          John
          http://john-goodwin.blogspot.com/
          • 2. Re: ESSSVR.exe crashes after service start
            974378
            Good morning John, and thanks for caring.

            I had already checked the essbase.log file. It shows nothing unusual.
            Following your advice I also checked SharedServices_Security_Client.log, but still nothing.

            Some research made me think that maybe I should set up the ESSBASEPATH and ARBORPATH variables in the System Variables of my Essbase server, which I did.
            After that I restarted services (the "OPMN service for Essbase" is the only Essbase service running on this server), but nothing happens.
            Then I tried restarting the server, but the problem persists.

            Is there anything else I could check to provide some clues about what's happening?

            Thank you.

            Regards,

            GSFeliu
            • 3. Re: ESSSVR.exe crashes after service start
              JohnGoodwin
              You shouldnt need to set the environment variables as they should be contained in the OPMN configuration file.
              You could also check the OPMN log files in <MIDDLEWARE_HOME>\user_projects\<instancename>\diagnostics\logs\OPMN\opmn

              Cheers

              John
              http://john-goodwin.blogspot.com/
              • 4. Re: ESSSVR.exe crashes after service start
                974378
                Thank you, John.

                Looking at the OPMN logs, I can see the following the problem started after a scheduled service restart. Prior to it and up to 11-20-2012 at 4:00 AM, everything was fine. When the service was automatically restarted at 4:40 AM after perfoming an automated back up (which is performed daily), the logs started to show errors and warnings. I pasted them here --maybe you can give me some ideas:

                opmn.log (nothing odd appears to be here)
                [2012-11-20T04:01:56][opmn][TRACE:1][668][OPMN]
                Request 3 Completed. Command: /shutdown
                [2012-11-20T04:01:56][opmn][TRACE:1][][OPMN][code:pm-requests]Http 3 
                EPM_epmsystem3~Essbase1~EssbaseAgent~AGENT~161747104:6416
                  Status: Stopped
                  Operation: request (time out while waiting for a managed process to stop)
                  ErrFile: C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
                  String: second stop attempted
                EPM_epmsystem3~Essbase1~EssbaseAgent~AGENT~161747104:6416
                  Status: Stopped
                  Operation: request (success)
                  ErrFile: C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
                
                [2012-11-20T04:01:56][opmn][TRACE:1][666][OPMN][code:pm-process]Process Stopped: Essbase1~EssbaseAgent~AGENT~1 (161747104:6416)
                [2012-11-20T04:40:15][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
                [2012-11-20T04:40:15][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: C:\Oracle\Middleware\user_projects\epmsystem3\config\OPMN\opmn\states
                [2012-11-20T04:40:15][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.
                [2012-11-20T04:40:15][opmn][TRACE:1][667][OPMN][code:pm-requests]Request 2 Started.  Command: /start
                [2012-11-20T04:40:15][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
                [2012-11-20T04:40:35][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
                [2012-11-20T04:40:46][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
                [2012-11-20T04:40:54][opmn][TRACE:1][668][OPMN][code:pm-requests]Request 2 Completed. Command: /start
                
                *service.log*
                --------
                12/11/20 04:00:48 shutdown
                --------
                
                --------
                12/11/20 04:40:16 startproc
                --------
                ================================================================================
                opmn id=DESHYPDB02.codere.com:6712
                    0 of 1 processes started.
                *
                ias-instance id=EPM_epmsystem3
                ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                --------------------------------------------------------------------------------
                ias-component/process-type/process-set:
                    Essbase1/EssbaseAgent/AGENT/
                
                Error
                --> Process (index=1,uid=161747105,pid=6612)
                    failed to start a managed process after the maximum retry limit
                    Log:
                    C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
                
                
                *console~Essbase1~EssbaseAgentÃGENT~1.log* (I copy what seems to be the most relevant information)
                Read Error:Fatal Error. C:\Oracle\Middleware\user_projects\epmsystem3\EssbaseServer\essbaseserver1\bin\*TEMPESSBASE.SEC is corrupt*
                
                Fatal Error: Essbase did not shutdown cleanly. Please remove TEMPESSBASE.SEC and restart Essbase with the ESSBASE.SEC file. Some security changes could have been lost
                
                According to this last error, shall I just remove (or back up) the TEMPESSBASE.SEC file, leave ESSBASE.SEC and try a restart?
                
                Many thanks,
                
                G.Sanchez                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                • 5. Re: ESSSVR.exe crashes after service start
                  JohnGoodwin
                  Yes try removing tempessbase.sec or if that doesnt work try renaming essbase.bak to essbase.sec or revert from a backup of essbase.sec

                  Cheers

                  John
                  http://john-goodwin.blogspot.com/
                  • 6. Re: ESSSVR.exe crashes after service start
                    974378
                    Hi again, John.

                    Thanks for your help. It really worked and Essbase.exe is now running as well as the Essbase applications.
                    However, we are facing a second problem and maybe you can guide us here, too.

                    Now, the development EAS console freezes up when you try to open the Essbase server node. This does not happen when you use the production console, from where we are able to see both the development and the production Essbase servers.

                    It comes to my mind that this could be related to user credentials. For security reasons within the company the admin password was changed yesterday, but it was eventually reverted to the previous password.
                    We have several admin accounts, but all of them experience the same issue.

                    Thank you.

                    Regards,

                    G.Sanchez
                    • 7. Re: ESSSVR.exe crashes after service start
                      974378
                      Ok, forget about it.

                      I restarted services in all my Hyperion servers and now everything is fine.

                      Thanks a lot.