10 Replies Latest reply: Nov 8, 2012 6:25 AM by YSP RSS

    Hyperion Services are getting stopped

    YSP
      Hi All,

      This has happened so many times that only restarting the hyperion services solved the issue.
      I have checked logs many times but dint get any clue.
      Yesterday I faced the same.
      I have got the complaint from client that Planning application is not working.
      I just checked that services of Foundation services and planning services were down.
      I just started those again and it worked.

      The log of planning had this information
      2012-11-01 21:10:13,463 WARN [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' com.hyperion.hbr.db.DBLocationTable - Unable to find Location with ID: 0



      2012-11-05 00:10:47,724 WARN Timer-3 com.hyperion.hbr.db.BaseRepository - Error while executing Database update: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:10:47,724 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:10:47,724 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: insert into HBRPluginData (PlanningServer, PlanningRmiPort, PlanningAppName, OlapServerName, OlapUserName, OlapPassword) values(?, ?, ?, ?, ?, ?)
      2012-11-05 00:10:47,724 FATAL Timer-3 com.hyperion.hbr.core.PluginDataManager - Error adding Plugin Data.
      2012-11-05 00:10:54,635 WARN Thread-37 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: select PlanningServer, PlanningRmiPort, PlanningAppName, OlapServerName, OlapUserName, OlapPassword, EntryTime from HBRPluginData where EntryTime > ?
      2012-11-05 00:10:54,635 FATAL Thread-37 com.hyperion.hbr.core.PluginDataManager - Error reading Plugin Data.
      2012-11-05 00:11:12,732 WARN Timer-3 com.hyperion.hbr.db.BaseRepository - Error while executing Database update: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:11:12,732 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:11:12,732 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: insert into HBRPluginData (PlanningServer, PlanningRmiPort, PlanningAppName, OlapServerName, OlapUserName, OlapPassword) values(?, ?, ?, ?, ?, ?)
      2012-11-05 00:11:12,732 FATAL Timer-3 com.hyperion.hbr.core.PluginDataManager - Error adding Plugin Data.
      2012-11-05 00:11:24,651 WARN Thread-37 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: select PlanningServer, PlanningRmiPort, PlanningAppName, OlapServerName, OlapUserName, OlapPassword, EntryTime from HBRPluginData where EntryTime > ?
      2012-11-05 00:11:24,651 FATAL Thread-37 com.hyperion.hbr.core.PluginDataManager - Error reading Plugin Data.
      2012-11-05 00:11:37,741 WARN Timer-3 com.hyperion.hbr.db.BaseRepository - Error while executing Database update: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:11:37,741 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: delete from HBRPluginData where UPPER(PlanningServer) = 'HYPERIONAPP' and UPPER(PlanningAppName) = 'PSSL' and UPPER(PlanningRmiPort) = '11333'
      2012-11-05 00:11:37,741 WARN Timer-3 com.hyperion.hbr.db.DBPluginDataTable - Error while executing query: insert into HBRPluginData (PlanningServer, PlanningRmiPort, PlanningAppName, OlapServerName, OlapUserName, OlapPassword) values(?, ?, ?, ?, ?, ?)
      2012-11-05 00:11:37,741 FATAL Timer-3 com.hyperion.hbr.core.PluginDataManager - Error adding Plugin Data.
      2012-11-05 00:11:55,666 WARN Thread-37 com.hyperion.hbr.db.DBConnectionManager - Unable to create a new connection for jdbc:oracle:thin:@pep-prd-dbs.shopzone.com:1521:HRPROD
      2012-11-05 00:11:55,666 FATAL Thread-37 com.hyperion.hbr.core.PluginDataManager - Error reading Plugin Data.
      2012-11-05 00:12:03,747 WARN Timer-3 com.hyperion.hbr.db.DBConnectionManager - Unable to create a new connection for jdbc:oracle:thin:@pep-prd-dbs.shopzone.com:1521:HRPROD
      2012-11-05 00:12:03,747 FATAL Timer-3 com.hyperion.hbr.core.PluginDataManager - Error adding Plugin Data.
      2012-11-05 00:12:04,777 WARN Thread-35 com.hyperion.hbr.db.DBConnectionManager - Unable to create a new connection for jdbc:oracle:thin:@pep-prd-dbs.shopzone.com:1521:HRPROD

      2012-11-05 11:40:14,059 WARN [ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' com.hyperion.hbr.security.HbrSecurityAPI - Error retrieving user by identity




      At 11:30, Nov 6, I started the services and it worked fine.

      Please let me know the reasons behind that.
        • 1. Re: Hyperion Services are getting stopped
          JohnGoodwin
          OS, version may help a little, which log have you posted - filename and location.

          Cheers

          John
          http://john-goodwin.blogspot.com/
          • 2. Re: Hyperion Services are getting stopped
            YSP
            Hi,

            The OS is Windows server 2008, 64 bit.
            The file name is "hbrserver" from path Diagnostics->Logs-> Planning.

            In past, we have seen a lot of issues related to relational database.
            This time again it might be related to the same, but we are not getting any clue as services are working fine once we start it up after getting down.

            Thanks
            Yashwant

            Edited by: yash13491012 on Nov 6, 2012 1:21 AM
            • 3. Re: Hyperion Services are getting stopped
              JohnGoodwin
              That is the business rules log, if you experience issues then it is probably worth checking the two planning logs in the diagnostics\logs\services directory, when you restart the service it will clear this log so each look at the log before you restart or take a copy, you can also check the weblogic server logs in <MIDDLEWARE_HOME>\user_projects\domains\EPMSystem\servers\Planning0\logs

              Cheers

              John
              http://john-goodwin.blogspot.com/
              • 4. Re: Hyperion Services are getting stopped
                YSP
                Hi,

                I checked the planning logs on the specified path and got the foloowing information.
                The following information was repeated in the logs for around 12 hours and after that, we had restarted the services.




                ####<Nov 5, 2012 3:21:01 AM IST> <Warning> <JDBC> <HYPERIONAPP> <Planning0> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1352065861644> <BEA-001129> <Received exception while creating connection for pool "planning_datasource": The Network Adapter could not establish the connection>
                ####<Nov 5, 2012 3:21:01 AM IST> <Info> <JDBC> <HYPERIONAPP> <Planning0> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1352065861644> <BEA-001156> <Stack trace associated with message 001129 follows:

                java.sql.SQLException: The Network Adapter could not establish the connection
                at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
                at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:480)
                at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:413)
                at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
                at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
                at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
                at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:320)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:220)
                at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
                at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2387)
                at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

                Caused By: oracle.net.ns.NetException: The Network Adapter could not establish the connection
                at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:328)
                at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:421)
                at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:634)
                at oracle.net.ns.NSProtocol.connect(NSProtocol.java:208)
                at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
                at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
                at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
                at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
                at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
                at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:320)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:220)
                at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
                at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2387)
                at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

                Caused By: java.net.ConnectException: Connection refused: connect
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:519)
                at java.net.Socket.connect(Socket.java:469)
                at java.net.Socket.<init>(Socket.java:366)
                at java.net.Socket.<init>(Socket.java:180)
                at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:127)
                at oracle.net.nt.ConnOption.connect(ConnOption.java:126)
                at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:306)
                at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:421)
                at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:634)
                at oracle.net.ns.NSProtocol.connect(NSProtocol.java:208)
                at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
                at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
                at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
                at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
                at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
                at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:320)
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:220)
                at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
                at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2387)
                at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
                • 5. Re: Hyperion Services are getting stopped
                  JohnGoodwin
                  These suggests there may be an issue connecting to the planning system relational database

                  *<Received exception while creating connection for pool "planning_datasource": The Network Adapter could not establish the connection>*

                  java.sql.SQLException: The Network Adapter could not establish the connection

                  Cheers

                  John
                  http://john-goodwin.blogspot.com/
                  • 6. Re: Hyperion Services are getting stopped
                    YSP
                    Hi,

                    Thanks for the information.
                    Please let me know the possible reasons for the same asand the procedure to know the exact reason from them.

                    Thanks
                    Yashwant
                    • 7. Re: Hyperion Services are getting stopped
                      YSP
                      Hi, I am still awaiting for your response.
                      • 8. Re: Hyperion Services are getting stopped
                        JohnGoodwin
                        There are issues connecting to the planning system relational database, it could be down, there could be a firewall or network issues, the connection information could be incorrect.

                        Cheers

                        John
                        http://john-goodwin.blogspot.com/
                        • 9. Re: Hyperion Services are getting stopped
                          YSP
                          Thanks for your quick reply.
                          As per my knowledge connection information are not wrong because when we restart the services it goes up.
                          And as they goes up successfully it means database is also up and firewall is also not an issue.

                          Now to get the exact problem, please let me know which type of logs should I look into.

                          Thanks
                          Yashwant
                          • 10. Re: Hyperion Services are getting stopped
                            YSP
                            Hi,

                            I am expecting some reply from you experts.
                            Please suggest.

                            Thanks
                            Yashwant