2 Replies Latest reply: Feb 22, 2013 4:24 AM by EdSpn RSS

    Agent blocked!

    EdSpn
      Hi,

      I started testing with discovery in my plug-in today, but now there seems to be an issue due to plugin version mismatch (see log messages below).
      In my deploy to test discovery I deployed the plugin on the server and ran the discovery UI in OEM. I didn't deploy to the agent that time.
      Later I wanted to redeploy a new version and tried to deploy to the server again, and from here it started failing.

      Below is part of the logfile ($OMS_HOME/sysman/log) which I think indicates the root cause.
      2013-02-21 10:51:17,461 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR gcloader.Receiver
       logp.251 - The plugin, [ The plug-in is emc.oem.star
       Version in pla inventory : 12.1.0.5.0
       Version on agent : 12.1.0.6.0
       Content type : AGENT
       Error code : MISSING_DISCOVERY_INPUT_WHEN_APPLICABLE
       Error Msg :  Discovery is applicable for this plug-in but has not been provided in the input] from agent:https://dwbi-durl-oel.
      gso.lab.emc.com:3872/emd/main/ does not match with repository
      2013-02-21 10:51:17,761 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR gcloader.Receiver
       logp.251 - Agent:https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ was blocked due to plugin match between repository and ag
      ent
      2013-02-21 10:51:17,776 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR gcloader.Util log
      p.251 - [EM-03554]  The plugin, [ The plug-in is emc.oem.star
       Version in pla inventory : 12.1.0.5.0
       Version on agent : 12.1.0.6.0
       Content type : AGENT
       Error code : MISSING_DISCOVERY_INPUT_WHEN_APPLICABLE
       Error Msg :  Discovery is applicable for this plug-in but has not been provided in the input] from agent https://dwbi-durl-oel.
      gso.lab.emc.com:3872/emd/main/ does not match with repository. Agent was blocked due to this match
      java.lang.Exception: The plugin, [ The plug-in is emc.oem.star
       Version in pla inventory : 12.1.0.5.0
       Version on agent : 12.1.0.6.0
       Content type : AGENT
       Error code : MISSING_DISCOVERY_INPUT_WHEN_APPLICABLE
       Error Msg :  Discovery is applicable for this plug-in but has not been provided in the input] from agent:https://dwbi-durl-oel.
      gso.lab.emc.com:3872/emd/main/ does not match with repository
              at oracle.sysman.core.pbs.gcloader.Receiver.doGet(Receiver.java:932)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
              :
      I have tried a number of things, but can't get out of this situation, I must be missing something...
      [oracle@dwbi-durl-oel ~]$ emcli undeploy_plugin_from_server -plugin="emc.oem.star" -sys_password="$SYSMANPWD"
      Plug-in emc.oem.star can not be un-deployed because:
      Plugin EMC Storage Array Plug-In:12.1.0.6.0 is deployed on 1 host, it can not be undeployed.
      Management Agent for Management Service dwbi-durl-oel.gso.lab.emc.com:4890_Management_Service is not UP. Can not undeploy EMC Storage Array Plug-In plugin
      
      [oracle@dwbi-durl-oel ~]$ emcli list_plugins_on_server | grep -i "Storage Array Plug-In"
      EMC Storage Array Plug-In                                                                           12.1.0.6.0
      [oracle@dwbi-durl-oel ~]$ emcli list_plugins_on_agent | grep -i "Storage Array Plug-In"
      EMC Storage Array Plug-In                                                                           12.1.0.5.0
      
      [oracle@dwbi-durl-oel plugin_ed3]$ emcli deploy_plugin_on_agent -agent_names=dwbi-durl-oel.gso.lab.emc.com:3872 -plugin=emc.oem.star:12.1.0.6.0
      Pre-requisites check failed
      For agent : https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/
              Plugin deployment on Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ cannot proceed because Master Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ is down. Start the Master Agent and retry plugin deployment
      
      [oracle@dwbi-durl-oel plugin_ed3]$ $AGENT_HOME/bin/emctl start agent
      Oracle Enterprise Manager Cloud Control 12c Release 2
      Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
      Agent is already running
      So I need to lift the Agent block, but don't know how..

      Any suggestions highly appreciated.

      Thanks,
      Ed
        • 1. Re: Agent blocked!
          EdSpn
          Just got some more log messages from $OMS_HOME/sysman/log.
          It says to start the master Agent there isn't any documentation on that?
          2013-02-21 18:42:00,028 [EMUI_18_41_59_/console/cli] ERROR agent.EMPluginAgentDeploymentHelper logp.251 - Plugin deployment on Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ cannot proceed because Master Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ is down. Start the master Agent and retry plugin deployment
          2013-02-21 18:42:00,095 [EMUI_18_41_59_/console/cli] ERROR agent.EMPluginAgentDeploymentHelper logp.251 - Plugin deployment on Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ cannot proceed because Master Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ is down. Start the master Agent and retry plugin deployment
          2013-02-21 18:49:59,778 [[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR agent.EMPluginAgentDeploymentHelper logp.251 - Plugin deployment on Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ cannot proceed because Master Agent https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ is down. Start the master Agent and retry plugin deployment
          2013-02-21 18:49:59,788 [[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN  resync.DiscoveryResyncCallback logp.251 - [On agent: https://dwbi-durl-oel.gso.lab.emc.com:3872/emd/main/ Plugin: oracle.sysman.oh failed with version: 12.1.0.2.0 Reason: Plugin deployment on Agent {0} cannot proceed because Master Agent {1} is down. Start the Master Agent and retry plugin deployment]
          2013-02-21 18:49:59,879 [[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN  oms.TMRemoteClient logp.251 - invalid quiesce timeout (0) specified: using 86400 instead
          2013-02-21 18:50:00,060 [[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN  resync.DiscoveryResyncCallback logp.251 - Problem discovery resynchronization : emd_guid and host_guid information missed for plug_id:oracle.sysman.oh
          Thanks,
          Ed
          • 2. Re: Agent blocked!
            EdSpn
            Update: Issue solved?

            It looks like the agent got blocked as there was still a target instance associated with this plugin. After deleting of the target I was able to undeploy from server and agent. The plugin is now deploying again :)
            [oracle@dwbi-durl-oel log]$ emcli delete_target -name="VMAX Symm IP.195" -type=storage_array_symm
            Target "VMAX Symm IP.195:storage_array_symm" deleted successfully
            
            [oracle@dwbi-durl-oel log]$ emcli undeploy_plugin_from_server -plugin="emc.oem.star" -sys_password="$SYSMANPWD"
            Plug-in emc.oem.star can not be un-deployed because:
            Plugin EMC Storage Array Plug-In:12.1.0.6.0 is deployed on 1 host, it can not be undeployed.
            
            [oracle@dwbi-durl-oel log]$ emcli undeploy_plugin_from_agent -plugin="emc.oem.star" -agent_names="dwbi-durl-oel.gso.lab.emc.com:3872"
            Plug-in undeployment from the management agents is in progress
            Use "emcli get_plugin_deployment_status -plugin_id=emc.oem.star" to track plug-in undeployment status.
            
            [oracle@dwbi-durl-oel log]$ emcli get_plugin_deployment_status -plugin_id=emc.oem.star |grep ^Status
            Status               : Success
            
            [oracle@dwbi-durl-oel log]$ emcli undeploy_plugin_from_server -plugin="emc.oem.star" -sys_password="$SYSMANPWD"
            Undeployment of plug-in from the management servers is in progress
            Use " emcli get_plugin_deployment_status -plugin_id=emc.oem.star " to track plug-in un-deployment status.
            
            [oracle@dwbi-durl-oel log]$ emcli get_plugin_deployment_status -plugin_id=emc.oem.star |grep ^Status
            Status               : Success
            Regards,
            Ed