1 2 Previous Next 23 Replies Latest reply: Apr 12, 2010 4:37 AM by Rajesh Lathwal RSS

    emctl start agent ...... Failed

    765364
      Hi,
      I am a Junior Oracle DBA and need help with an issue I am having with the OEM agent please.

      When I run the command emctl start agent i get the below message.

      Starting agent ................................. failed.

      I have already tried the following as seen on other forums:

      1. Changing the port number in the emd.properties file to 3873 - Didnt work
      2. Changing the AgentListenOnAllNICs in the emd.properties file to be FALSE - Didnt work
      3. Removing all files from agent10g/sysman/emd/state/ and agent10g/sysman/emd/upload/ and running emctl clearstate - Didnt work

      Any help on this would be greatly appreciated. Thanks. Below are the log files.


      emdctl.trc
      =======

      2010-04-08 10:21:14 Thread-4136921312 WARN http: snmehl_connect: connect failed to (db3.domain.com:3872): Network is unreachable (error = 101)
      2010-04-08 10:21:14 Thread-4136921312 ERROR main: nmectla_agentctl: Error connecting to http://db3.domain.com:3872/emd/main/. Returning status code 1
      2010-04-08 10:21:44 Thread-4136921312 WARN http: snmehl_connect: connect failed to (db3.domain.com:3872): Network is unreachable (error = 101)
      2010-04-08 10:21:44 Thread-4136921312 ERROR main: nmectla_agentctl: Error connecting to http://db3.domain.com:3872/emd/main/. Returning status code 1
      2010-04-08 10:22:14 Thread-4136921312 WARN http: snmehl_connect: connect failed to (db3.domain.com:3872): Network is unreachable (error = 101)
      2010-04-08 10:22:14 Thread-4136921312 ERROR main: nmectla_agentctl: Error connecting to http://db3.domain.com:3872/emd/main/. Returning status code 1
      2010-04-08 10:22:44 Thread-4136921312 WARN http: snmehl_connect: connect failed to (db3.domain.com:3872): Network is unreachable (error = 101)
      2010-04-08 10:22:44 Thread-4136921312 ERROR main: nmectla_agentctl: Error connecting to http://db3.domain.com:3872/emd/main/. Returning status code 1
      2010-04-08 10:23:14 Thread-4136921312 WARN http: snmehl_connect: connect failed to (db3.domain.com:3872): Network is unreachable (error = 101)
      2010-04-08 10:23:14 Thread-4136921312 ERROR main: nmectla_agentctl: Error connecting to http://db3.domain.com:3872/emd/main/. Returning status code 1

      emagent.nohup
      ===========

      ----- Thu Apr 8 10:21:14 2010::Checking status of EMAgent : 4693 -----
      sh: line 1: 5919 Aborted (core dumped) /opt/oracle/product/10.2.0/agent10g/bin/emdctl status agent 300 >/dev/null 2>&1
      ----- Thu Apr 8 10:21:44 2010::Checking status of EMAgent : 4693 -----
        • 1. Re: emctl start agent ...... Failed
          Rob Zoeteweij
          Any other log, trc files written?
          Go to <AGENT_HOME>/sysman/log and do: ls -ltr
          Take a look in the other output files for any clues.

          You might have a Time Zone issue.

          Regards
          Rob
          • 2. Re: emctl start agent ...... Failed
            765364
            The timezone in the emd.properties file is agentTZRegion=GMT
            The timezone in the emd.properties file on the server where OEM is installed is agentTZRegion=Europe/Dublin


            The only log file written to at the time would be emagent.trc which I have pasted below.

            emagent.trc
            =========
            2010-04-08 10:12:12 Thread-4121516976 WARN scheduler: oracle_emd:db3.domain.com:3872:ProcessInfo is due to run again but is still running from last schedule
            2010-04-08 10:29:30 Thread-4121516976 WARN scheduler: oracle_emd:db3.domain.com:3872:ProcessInfo is due to run again but is still running from last schedule
            2010-04-08 10:29:31 Thread-4075785136 ERROR fetchlets.oslinetok: Metric execution timed out in 2076 seconds
            2010-04-08 10:29:33 Thread-4075785136 ERROR fetchlets.oslinetok: Process stderr = ERROR: Process ID list syntax error.
            ********* simple selection ********* ********* selection by list *********
            -A all processes -C by command name
            -N negate selection -G by real group ID (supports names)
            -a all w/ tty except session leaders -U by real user ID (supports names)
            -d all except session leaders -g by session OR by effective group name
            -e all processes -p by process ID
            T all processes on this terminal -s processes in the sessions given
            a all w/ tty, including other users -t by tty
            g OBSOLETE -- DO NOT USE -u by effective user ID (supports names)
            r only running processes U processes for specified users
            x processes w/o controlling ttys t by tty
            *********** output format ********** *********** long options ***********
            -o,o user-defined -f full Group User pid cols --ppid
            -j,j job control s signal group user sid rows --info
            -O,O preloaded -o v virtual memory cumulative format --deselect
            -l,l long u user-oriented sort tty forest version
            -F extra full X registers heading no-heading --context
            ********* misc options *********
            -V,V show version L list format codes f ASCII art forest
            -m,m,-L,-T,H threads S children in sum -y change -l format
            -M,Z security data c true command name -c scheduling class
            -w,w wide output n numeric WCHAN,UID -H process hierarchy

            2010-04-08 10:29:33 Thread-4075785136 ERROR engine: [oracle_emd,db3.domain.com:3872,ProcessInfo] : nmeegd_GetMetricData failed : Metric execution timed out in 2076 seconds
            2010-04-08 10:29:33 Thread-4075785136 WARN collector: <nmecmc.c> Error exit. Error message: Metric execution timed out in 2076 seconds
            2010-04-08 10:35:39 Thread-4075785136 ERROR fetchlets.oslinetok: Process stdout = em_error=failed to get upload statistics:
            2010-04-08 10:35:39 Thread-4075785136 ERROR fetchlets.oslinetok: Process stderr = Error connecting to http://db3.domain.com:3872/emd/main/

            2010-04-08 10:35:39 Thread-4075785136 ERROR engine: [oracle_emd,db3.domain.com:3872,EMDUploadStats] : nmeegd_GetMetricData failed : em_error=failed to get upload statistics:
            Error connecting to http://db3.domain.com:3872/emd/main/

            2010-04-08 10:35:39 Thread-4075785136 WARN collector: <nmecmc.c> Error exit. Error message: em_error=failed to get upload statistics:
            Error connecting to http://db3.domain.com:3872/emd/main/
            2010-04-08 11:04:06 Thread-4121516976 WARN scheduler: oracle_emd:db3.cdomain.com:3872:ProcessInfo is due to run again but is still running from last schedule

            Edited by: user5763760 on 08-Apr-2010 04:21
            • 3. Re: emctl start agent ...... Failed
              Rob Zoeteweij
              What timezone is displayed when you type: date

              Time zone issues can be debugd as described in How to Troubleshoot Timezone Issues in Enterprise Manager 10g [ID 330737.1] on My Oracle Support
              • 4. Re: emctl start agent ...... Failed
                765364
                The timezone on the target server is GMT (Its GMT in the emd.properties file).

                The timezone for the server hosting OEM is IST (Its Europe/Dublin in the emd.properties file).

                Is it the fact that the server TZ doesnt match with that in the emd.properties file?
                • 5. Re: emctl start agent ...... Failed
                  Rob Zoeteweij
                  cd <AGENT_HOME>/sysman/config
                  grep agentTZ emd.properties
                  $ EMDROOT=$ORACLE_HOME
                  $ EMSTATE=$ORACLE_HOME
                  $ export EMDROOT EMSTATE
                  emdctl validateTZ agent <TZ region from emd.properties>

                  If no message is shown, you ruled out a time zone issue.
                  Otherwise comment out the agentTZregion variable from the emd.properties file
                  restart the agent and see what entry was added to the emd.properties file

                  let me know the outcome of your test
                  • 6. Re: emctl start agent ...... Failed
                    Rajesh Lathwal
                    Hi,

                    Did you try unsecuring the agent and then try to start it..

                    emctl unsecure agent
                    emctl start agent..
                    • 7. Re: emctl start agent ...... Failed
                      765364
                      Rob, I followed your steps and nothing was returned when the emdctl validateTZ agent GMT command.

                      Rajesh, I tried to run emctl unsecure agent but it doesnt recognise unsecure as a valid option.

                      $ emctl unsecure agent
                      Unknown command option unsecure
                      Usage::
                      Oracle Enterprise Manager 10g Database Control commands:
                      emctl start| stop| status| setpasswd dbconsole
                      emctl secure <options>

                      Thanks for your ongoing help with this guys.

                      Rob
                      • 8. Re: emctl start agent ...... Failed
                        Rob Zoeteweij
                        It seems that you are working on Enterprise Manager Database Control
                        is this what you wre expecting?
                        If you are working on an Enterprise Manager Grd Control environment, you should first set ORACLE_HOME to your Agent environment.
                        This should normally be something like /u01/app/oracle/product/10.2.0/agent10
                        Here in the <AGENT_HOME>/bin you should execute emctl
                        -
                        What you are doing now, is executing this command from the Database Oracle Home.
                        -
                        So, what is it you intend to work with?
                        Local Database control using Database Control or
                        GRID COntrol being able to manager all your databases (and more)

                        Regards
                        Rob
                        • 9. Re: emctl start agent ...... Failed
                          765364
                          I am looking to work with Grid Control.

                          I have set ORACLE_HOME to /opt/oracle/product/10.2.0/agent10g and I have been running the emctl from the bin directory. But still no joy :-(
                          • 10. Re: emctl start agent ...... Failed
                            Rob Zoeteweij
                            Didi you retry all sugestions:
                            secure agent
                            check time zone
                            using the correct $ORACLE_HOME
                            -
                            You lost feedback pointed out that you were refering to the wrong Oracle home
                            • 11. Re: emctl start agent ...... Failed
                              765364
                              Yeah, I tried all of the above with the new Oracle_Home but still getting the same result. Still the same errors in the logs too.

                              [oracle@db3 bin]$ pwd
                              /opt/oracle/product/10.2.0/agent10g/bin
                              [oracle@db3 bin]$ echo $ORACLE_HOME
                              /opt/oracle/product/10.2.0/agent10g
                              [oracle@db3 bin]$ emctl status agent
                              Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
                              Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
                              ---------------------------------------------------------------
                              Agent is Not Running
                              [oracle@db3 bin]$ emctl start agent
                              Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
                              Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
                              Starting agent ................................. failed.
                              • 12. Re: emctl start agent ...... Failed
                                Rob Zoeteweij
                                How about securing the agent?
                                • 13. Re: emctl start agent ...... Failed
                                  765364
                                  I got the following error when trying the secure agent.

                                  [oracle@db3 bin]$ emctl secure agent
                                  Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
                                  Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
                                  Enter Agent Registration password :
                                  Requesting an HTTPS Upload URL from the OMS...caught ParseException HTTPClient.ParseException: %EM_HTTP_PORT% is an invalid port number
                                  Failed.
                                  The OMS is not set up for Enterprise Manager Security.
                                  [oracle@db3 bin]$
                                  • 14. Re: emctl start agent ...... Failed
                                    Rob Zoeteweij
                                    Reset your portnumber to 3872 (emd.properties)
                                    Clear all log files (<AGENT_HOME>/sysman/log)
                                    start agent
                                    Start agent (<AGENT_HOME>/bin/emctl start agent)
                                    check the log and trc files
                                    1 2 Previous Next