8 Replies Latest reply: Sep 21, 2007 5:47 AM by 277993 RSS

    Agent is Running but Not Ready

    232885
      Hi

      One of my development server oracle 10g agent is not showing up in Grid control. I have done a clean restart of oracle agent with this process

      1. Stop target agent host
      <AGENT_HOME>/bin/emctl stop agent

      2. On Verify no emagent process are running
      ps -ef|grep emagent

      3. Clean up old residue files
      cd <AGENT_HOME>/sysman/emd
      rm -rf agntstmp.txt lastupld.xml recv/* collection/* upload/* state/*

      4. Secure Agent
      <AGENT_HOME>/bin/emctl secure agent

      5. Start Agent
      <AGENT_HOME>/bin/emctl start agent

      6. Set Clearstate for Agent
      <AGENT_HOME>/bin/emctl clearstate agent

      7. Upload data for agent
      <AGENT_HOME>/bin/emctl upload agent

      8. Run status for agent
      <AGENT_HOME>/bin/emctl status agent

      agent is getting started and I am getting the following error.


      C:\Oracle\agent10g\BIN>emctl status agent
      Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
      Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
      ---------------------------------------------------------------
      Agent Version : 10.2.0.3.0
      OMS Version : 10.2.0.3.0
      Protocol Version : 10.2.0.2.0
      Agent Home : C:\Oracle\agent10g
      Agent binaries : C:\Oracle\agent10g
      Agent Process ID : 3932
      Agent URL : https://shadbdev7.shacadd.ad.mdot.mdstate:3872/emd/main/
      Repository URL : https://SHADBDEV6.sha.mdot.mdstate:1159/em/upload
      Started at : 2007-09-18 15:55:46
      Started by user : SYSTEM
      Last Reload : 2007-09-18 15:55:46
      Last successful upload : 2007-09-18 16:00:07
      Total Megabytes of XML files uploaded so far : 1.22
      Number of XML files pending upload : 8
      Size of XML files pending upload(MB) : 0.01
      Available disk space on upload filesystem : 44.42%
      Last successful heartbeat to OMS : 2007-09-18 15:58:30
      ---------------------------------------------------------------
      Agent is Running and Ready

      C:\Oracle\agent10g\BIN>emctl upload agent
      Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
      Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
      ---------------------------------------------------------------
      EMD upload completed successfully


      C:\Oracle\agent10g\BIN>emctl status agent
      Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
      Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
      ---------------------------------------------------------------
      Agent is Not Running

      I do not know why it is behaving like this. Looking forward to your help.

      Rao
        • 1. Re: Agent is Running but Not Ready
          277993
          You may be in the Agent directory, but you may not have the correct Home eset. you need to change ORACLE_HOME to the AGENT_HOME before you try any actions for the agent.

          C:\> set ORACLE_HOME=C:\Oracle\agent10g
          C:\> emctl status agent

          the same things need to be done when you want to do anything for the OMS and DB
          • 2. Re: Agent is Running but Not Ready
            232885
            Hi

            My issue is agent is getting stopped automatically.I have two oracle agents are running in this box with two different oracle homes. I am attaching the

            C:\>set oracle_home=C:\Oracle\agent10g

            C:\>emctl status agent
            Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
            Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
            ---------------------------------------------------------------
            Agent is Not Running

            C:\>emctl start agent
            Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
            Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
            The Oracleagent10g1Agent service is starting........................
            The Oracleagent10g1Agent service was started successfully.


            C:\>emctl status agent
            Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
            Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
            ---------------------------------------------------------------
            Agent Version : 10.2.0.3.0
            OMS Version : 10.2.0.3.0
            Protocol Version : 10.2.0.2.0
            Agent Home : C:\Oracle\HANGRID\agent10g
            Agent binaries : C:\Oracle\HANGRID\agent10g
            Agent Process ID : 4280
            Agent URL : https://shadbdev7.shacadd.ad.mdot.mdstate:1830/emd/main/
            Repository URL : https://SHAHANOEMG.shacadd.ad.mdot.mdstate:1159/em/upload
            Started at : 2007-09-19 10:10:49
            Started by user : SYSTEM
            Last Reload : 2007-09-19 10:10:49
            Last successful upload : 2007-09-19 10:12:04
            Total Megabytes of XML files uploaded so far : 0.15
            Number of XML files pending upload : 1
            Size of XML files pending upload(MB) : 0.00
            Available disk space on upload filesystem : 44.38%
            Last successful heartbeat to OMS : 2007-09-19 10:11:43
            ---------------------------------------------------------------
            Agent is Running and Ready


            C:\>emctl upload agent
            Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
            Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
            ---------------------------------------------------------------
            EMD upload completed successfully

            C:\>emctl status agent
            Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
            Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
            ---------------------------------------------------------------
            Agent Version : 10.2.0.3.0
            OMS Version : 10.2.0.3.0
            Protocol Version : 10.2.0.2.0
            Agent Home : C:\Oracle\HANGRID\agent10g
            Agent binaries : C:\Oracle\HANGRID\agent10g
            Agent Process ID : 4280
            Agent URL : https://shadbdev7.shacadd.ad.mdot.mdstate:1830/emd/main/
            Repository URL : https://SHAHANOEMG.shacadd.ad.mdot.mdstate:1159/em/upload
            Started at : 2007-09-19 10:10:49
            Started by user : SYSTEM
            Last Reload : 2007-09-19 10:10:49
            Last successful upload : 2007-09-19 10:12:35
            Total Megabytes of XML files uploaded so far : 0.26
            Number of XML files pending upload : 1
            Size of XML files pending upload(MB) : 0.01
            Available disk space on upload filesystem : 44.38%
            Last successful heartbeat to OMS : 2007-09-19 10:12:44
            ---------------------------------------------------------------
            Agent is Running and Ready

            Then all of a sudden I get this error. I tried several time doing a clean restart of agent still I am getting the same error. Here is the emdctl.trc file information.
            Agent status response - <abort agent>, Agent in abnormal state
            2007-09-19 09:48:19 Thread-3360 WARN main: Agent status response - <abort agent>, Agent in abnormal state
            2007-09-19 09:48:19 Thread-5076 WARN main: Agent status response - <abort agent>, Agent in abnormal state
            2007-09-19 09:54:49 Thread-4176 WARN http: snmehl_connect: connect failed to (shadbdev7.shacadd.ad.mdot.mdstate:3872): No connection could be made because the target machine actively refused it.
            (error = 10061)
            2007-09-19 09:54:49 Thread-4176 ERROR main: nmectla_agentctl: Error connecting to https://shadbdev7.shacadd.ad.mdot.mdstate:3872/emd/main/. Returning status code 1
            2007-09-19 10:01:14 Thread-5816 WARN http: snmehl_connect: connect failed to (shadbdev7.shacadd.ad.mdot.mdstate:3872): No connection could be made because the target machine actively refused it.
            (error = 10061)
            2007-09-19 10:01:14 Thread-5816 ERROR main: nmectla_agentctl: Error connecting to https://shadbdev7.shacadd.ad.mdot.mdstate:3872/emd/main/. Returning status code 1
            2007-09-19 10:02:08 Thread-3956 WARN http: snmehl_connect: connect failed to (shadbdev7.shacadd.ad.mdot.mdstate:3872): No connection could be made because the target machine actively refused it.
            (error = 10061)
            2007-09-19 10:02:08 Thread-3956 ERROR main: nmectla_agentctl: Error connecting to https://shadbdev7.shacadd.ad.mdot.mdstate:3872/emd/main/. Returning status code 1


            agabend.log


            MAgent is Thrashing. Exiting watchdog
            EMAgent is Thrashing. Exiting watchdog


            Looking forward to your help here. Several Grid jobs are hanging here.

            Rao
            • 3. Re: Agent is Running but Not Ready
              277993
              Fromthe error, one of these may be the cause.

              1. If you have two agents on the box, you may have a port conflict between the two
              2. You may have port access problem due to fireware config blocking it.
              3. You have a secured OMS, but Agent is not secured or vise versa
              • 4. Re: Agent is Running but Not Ready
                Sal
                specify the full path of the AGENT_HOME/bin you are trying to work with

                %AGENT_HOME%/bin/emctl status agent
                • 5. Re: Agent is Running but Not Ready
                  mnazim-Oracle
                  Reboot the box
                  • 6. Re: Agent is Running but Not Ready
                    232885
                    Thanks mnazim. I have tried rebooting almost 3 times. I have fixed the issue here by removing one of the Oracle process. In this box 7 development databases and 2 oracle agents are running before. I have removed the second oracle agent and shutdown the one of the database which was not in use. After stopping thes two services my agent is pretty much stable and running fine. Is there any limitation of services to one server? Here is my server Harware configuration.

                    Hardware

                    System Configuration ProLiant DL380 G2
                    Hardware Provider Compaq
                    Number of CPUs 2
                    Memory Size (MB) 3776
                    Related Link Hardware Details

                    Operating System

                    Operating System Microsoft Windows Server 2003 Server 5.2 Service Pack 2 (32-bit)
                    Patches 14
                    Related Link Operating System Details

                    Your inputs will be highly appreciated.

                    Thanks
                    Rao
                    • 7. Re: Agent is Running but Not Ready
                      mnazim-Oracle
                      Thats Great.

                      Following Configuration is supported. it seems some intermittent behaviour.
                      • 8. Re: Agent is Running but Not Ready
                        277993
                        As I mentioned above, it could be related to port conflicts. It will happen again if you started the other Agent.

                        One hint is when installing a second agent, ensure that the first agent is running. This way, Oracle will not allocate the same ports to the second agent because it will be in use.