3 Replies Latest reply: May 8, 2013 1:02 PM by BillW RSS

    12c Agents are not starting when host is restarted

    BillW
      I have upgraded agents on many of our hosts to 12c and am finding that on several hosts the agents are not starting on reboot.

      I found note Metalink Note 1508453.1 and followed it and according to the note everything appears to be correct. The /etc/init.d/gcstartup file exists, the oragchomelist exists in the locations /etc and /x01/Ora_Base/product/agent12g/core/12.1.0.1.0/install and the contents of the oragchomelist file is -

      /usr/local/oracle/Ora_Base/product/agent12g/core/12.1.0.1.0:/usr/local/oracle/Ora_Base/product/agent12g/agent_inst

      in both locations. These files were created during the upgrade process. I upgraded from OEM 11g using the Enterprise Manager 12c upgrade console and the 2-System method.

      This morning I restarted one of the hosts in question at 08:30 and I see the following -

      in the gcagent.log file -
      2013-05-07 08:32:25,936 [28:F9C26A76] INFO - jetty: Graceful shutdown ContextHandler@2b76086d@2b76086d/,null
      2013-05-07 08:32:29,082 [28:F9C26A76] INFO - jetty: Shutdown hook complete
      and nothing after that

      in the gcagent_errors.log file -

      2013-05-07 08:30:23,689 [65678:GC.SysExecutor.22100 (oracle_listener:LISTENER_payne:Real-time collection Response)] ERROR - Collection item result upload failure oracle_listener.LISTENER_payne::Response
      oracle.sysman.gcagent.target.interaction.execution.RTMUploadOnDemand$UploadOnDemandException: Could not send some or all data over to repository: Failed to upload file: null (TIMEOUT)

      n the emdctlj.log file -

      2013-05-07 08:41:17,016 [main] INFO - EmdCtl Timezone = America/Los_Angeles
      2013-05-07 08:41:22,042 [main] INFO - Command: status agent
      2013-05-07 08:41:22,635 [main] INFO - unable to connect to the agent at https://payne.ucdavis.edu:3872/emd/lifecycle/main/ [Connection refused]
      oracle.sysman.emSDK.agent.comm.exception.ConnectException: unable to connect to the agent at https://payne.ucdavis.edu:3872/emd/lifecycle/main/ [Connection refused]

      However I can start the agent manually with ./emctl start agent

      I have looked at the log files an the sysman/log directory but am unable to figure out what is going on although I suspect that there is some configuration error somewhere. Some of these hosts are members of clusters but I don't know if that is relevant either. I'd appreciate any suggestions anyone might offer.

      Thank you.
      Bill Wagman
        • 1. Re: 12c Agents are not starting when host is restarted
          BillW
          One other thing I just noticed is that in the /etc/oratab files on hosts on which this seems to be occurring (I have not checked them all) there is an entry for the 11g agent which is not commented. A segment follows -

          cmenv:/usr/local/oracle/product/11.2.03/temp:N # line added by Agent
          # *:/usr/local/oracle/product/agent11g:N
          *:/usr/local/oracle/product/agent11g:N
          mydevel:/usr/local/oracle/product/11.2.0.3/dbhome_1:N # line added by Agent

          Is that as it should be? I see two options, either comment out that line or change it to point to *:/usr/local/oracle/product/agent12g:N. I must admit I am unclear as to why that entry is in the oratab as it is. Clarification would be helpful.

          Thank you.
          Bill Wagman
          • 2. Re: 12c Agents are not starting when host is restarted
            Akanksha Sheoran Kaler-Oracle
            Hi Will,

            Pls open an SR for this issue and you can also try the following on one agent :

            remove the entry from the /etc/oratab file
            try to reboot once again or just run this command as root user /etc/init.d/gcstartup start - this command should start the agent as it would do it on the m/c reboot

            I am still not sure if agent upgrade did anything or not so we need an SR open to check this through logs. Can you also dropn an email to me and Anandu (support) with the SR, i will request Anandu to work on your Sr as she handelled all your past Sr's and is aware of your env history as well.
            • 3. Re: 12c Agents are not starting when host is restarted
              BillW
              Akanksha,

              Again, thank you for your assistance. I have opened SR 3-7170100601 on this issue.

              As to the issue with the oratab file, last night a host was rebooted because of OS patching and it had the agent entry in the oratab file. On that host the agent started so again, I am confused.

              Thank you.
              Bill Wagman