2 Replies Latest reply: Apr 29, 2013 6:23 AM by EdSpn RSS

    Undeployment from server stalled in deleting metadata

    EdSpn
      Hi all,

      Yesterday I tried to undeploy a plugin, but it didn’t succeed, it stalled in one of the undeployment steps.

      This is something that has been done many times before, so it puzzles me as to what could be causing this.
      In $OMS_HOME/cfgtoollogs/pluginca/ I found 2 directories of the 2 times that I tried to undeploy: plugin_undeployment_406 and .._409.

      Sequence of events:
      emcli delete_target -name="RHEL55-DB_5412"     -type="$TARGET_TYPE_NAME" --> successful
      emcli undeploy_plugin_from_agent  -plugin="$PLUGINID" -agent_names="$AGENT_NAME"  --> successful
      emcli undeploy_plugin_from_server -plugin="$PLUGINID" -sys_password="$SYSMAN_PWD"
      The last step (undepl from server) stays ‘unchanged’ in status ‘Running’. Eventually I decide to restart the OMS and Agent, as I am not sure why it is stalled:
      emcli undeploy_plugin_from_server -plugin="$PLUGINID" -sys_password="$SYSMAN_PWD"
      The last step (undepl from server) stays ‘unchanged’ in status ‘Running’ again, for step “Delete Plugin’s metadata”:
      [oracle@dwbi-durl-oel pluginca]$ emcli get_plugin_deployment_status -plugin_id=emc.mon.vmax
      Plugin Deployment/Undeployment Status
      
      Destination          : OMS - dwbi-durl-oel.gso.lab.emc.com:4890_Management_Service
      Plugin Name          : EMC VMAX Array Monitoring .
      Version              : 12.1.0.1.0
      ID                   : emc.mon.vmax
      Content              : Plugin
      Action               : Undeployment
      Status               : Undeploying
      Steps Info:
      ---------------------------------------- ------------------------- ------------------------- ----------
      Step                                     Start Time                End Time                  Status
      ---------------------------------------- ------------------------- ------------------------- ----------
      Submit job for un-deployment             4/24/13 12:07:54 PM EDT   4/24/13 12:07:54 PM EDT   Success
      Initialize                               4/24/13 12:08:01 PM EDT   4/24/13 12:08:01 PM EDT   Success
      Validate plug-in home                    4/24/13 12:08:02 PM EDT   4/24/13 12:08:03 PM EDT   Success
      Delete plugin's metadata                 4/24/13 12:08:03 PM EDT   N/A                       Running
      
      ---------------------------------------- ------------------------- ------------------------- ----------
      [oracle@dwbi-durl-oel pluginca]$ date
      Wed Apr 24 14:19:57 EDT 2013
      I had a look at the undeployment logfiles in $OMS_LOG/ cfgtoollogs/pluginca/plugin_undeployment_406 (1st try to undeploy) and found:
      [oracle@dwbi-durl-oel plugin_undeployment_406]$ grep ERROR configplugin_deconfig_2013-04-24_11-32-24.trc
      2013-04-24 11:32:30,276 [main] ERROR pluginca.PluginCADriver - Method not implemented
      2013-04-24 11:32:30,290 [main] ERROR pluginca.PluginCADriver - EMSDKSnapshotParser. Method deleteMetadata is not implemented
      2013-04-24 11:32:30,292 [main] ERROR pluginca.PluginCADriver - EMSDKSnapshotLiveParser. Method deleteMetadata is not implemented
      2013-04-24 11:32:30,292 [main] ERROR pluginca.PluginCADriver - Method not implemented
      and the undeployment logfiles in $OMS_LOG/ cfgtoollogs/pluginca/plugin_undeployment_409 (2nd try to undeploy):
      :
      2013-04-24 12:08:03,505 [main] INFO  pluginca.PluginCADriver - Invoking MRS delete metadata callback.
      2013-04-24 12:08:03,505 [main] INFO  pluginca.PluginCADriver - Invoking MRS delete metadata callbacks.
      2013-04-24 12:08:03,505 [main] DEBUG util.PluginCAUtil - Writing the following to job system output: Invoking MRS delete metadata callbacks.
      2013-04-24 12:08:03,505 [main] DEBUG emdrep.jobs - JobEngine: output for step 47011
      2013-04-24 12:08:03,510 [main] DEBUG lock.EMPluginOMSDeploymentStepLock - Obtaining lock for step DELETE_METADATA for 409 : D3FD034E3498227231B9D1BD45C2BE7F
      2013-04-24 12:08:03,511 [main] DEBUG lock.EMPluginOMSDeploymentStepLock - Currently locked by: 406 : D3FD034E3498227231B9D1BD45C2BE7F
      2013-04-24 12:08:03,511 [main] DEBUG lock.EMPluginOMSDeploymentStepLock - waiting for 1 seconds more for the lock to get released
      2013-04-24 12:08:04,512 [main] DEBUG lock.EMPluginOMSDeploymentStepLock - waiting for 2 seconds more for the lock to get released
      :
      Clearly the 2nd try was no use as the first still has a lock somehow.

      Any suggestions how to get us out of this situation quickly would be much appreciated.

      Thanks,
      Ed
        • 1. Re: Undeployment from server stalled in deleting metadata
          EdSpn
          Update:

          While the undeployment of that plugin failed, I thought I would copy the plugin, change the plugin-id slightly and deploy that.
          However, I just found out that any+ plugin deployment is blocked by this hanging undeployment. The new, clean plugin deployment failed with below error messages in the OMS logfile:
          :
          2013-04-25 06:31:05,928 [EMUI_06_31_05_/console/cli] ERROR emcli.DeployPluginToOMSVerb logp.251 - EM-04047: The last plug-in undeployment attempt has failed
          for plug-in emc.mon.vmax:12.1.0.1.0:0. This plug-in must be undeployed completely before deploying any plug-in. Re-try undeploying this plug-in or Contact Or
          acle Support for further help.
          oracle.sysman.core.common.extensibility.plugin.deploymentservice.EMPluginDeploymentServerSideException: EM-04047: The last plug-in undeployment attempt has f
          ailed for plug-in emc.mon.vmax:12.1.0.1.0:0. This plug-in must be undeployed completely before deploying any plug-in. Re-try undeploying this plug-in or Cont
          act Oracle Support for further help.
                  at oracle.sysman.core.common.extensibility.plugin.deploymentservice.EMPluginDeploymentServerSideException$Builder.build(EMPluginDeploymentServerSideE
          xception.java:87)
                  at oracle.sysman.core.common.extensibility.plugin.deploymentservice.EMPluginDeploymentUtil.checkForDanglingUndepFailure(EMPluginDeploymentUtil.java:3
          185)
          :
          Help!

          Many thanks for any feedback,
          Ed
          • 2. Re: Undeployment from server stalled in deleting metadata
            EdSpn
            To have information complete and available for other people on the forum:

            This question was successfully solved via SR 3-7120586441 : OEM plugin: stuck in undeployment issue

            Regards,
            Ed