4 Replies Latest reply: Oct 29, 2012 7:25 AM by user12273962 RSS

    SAN fell over. Trying to recover my OVM pool.

    945503
      My SAN died and knocked out my OVM pool. It fell over when the OS disk that the SAN was running from died and couldn't be recovered. It was hosting a ZFS pool to OVM. The OS disk was running a version of the Oracle ZFS storage appliance.

      I've been trying to recover the situation by rebuilding the storage appliance OS and representing the ZFS pool to it, and then configuring the OS to reimport the pool and give the SAN the same iSCSI configuration as before.

      Unsurprisingly this isn't working seamlessly.
      Can anyone help?

      I suspect that the reinstall of the OS has OVM-manager thinking that it's not the SAN it's looking for, even though the configuration is the same and the data along with it.
      In any case the OVM pool cluster isn't working.
      I want to avoid blowing away all my data just because it won't reconnect.

      I get the following error simply trying to refresh the SAN. I see lots of "login denied" errors, even though the username and password has been set to what they were previously. :-



      Job Construction Phase
      ----------------------
      begin()
      Appended operation 'Storage Server Discover Capabilities' to object '0004fb000009000022077528059c32a0 (ZFS-SA)'.
      Appended operation 'Storage Server Discover Access Groups' to object '0004fb000009000022077528059c32a0 (ZFS-SA)'.
      Appended operation 'Storage Array Discover Info' to object '0004fb000009000022077528059c32a0 (ZFS-SA)'.
      Appended operation 'Storage Array Discover Storage Elements' to object '0004fb000009000022077528059c32a0 (ZFS-SA)'.
      Appended operation 'ISCSI Storage Array Discover Targets' to object '0004fb000009000022077528059c32a0 (ZFS-SA)'.
      Appended operation 'Adding a queued job operation' to object '0004fb0000190000de6c2bd2cd7d2d79 (Refresh Storage Layer)'.
      Appended operation 'Adding a queued job operation' to object '0004fb0000190000408d083065e062ff (Refresh Storage Layer)'.
      commit()
      Completed Step: COMMIT

      Objects and Operations
      ----------------------
      Object (CREATED): [ServerRefreshStorageLayer] 0004fb0000190000de6c2bd2cd7d2d79 (Refresh Storage Layer)
      Operation: Adding a queued job operation
      Object (IN_USE): [IscsiStorageArray] 0004fb000009000022077528059c32a0 (ZFS-SA)
      Operation: Storage Server Discover Capabilities
      Operation: Storage Server Discover Access Groups
      Operation: Storage Array Discover Info
      Operation: Storage Array Discover Storage Elements
      Operation: ISCSI Storage Array Discover Targets
      Object (CREATED): [ServerRefreshStorageLayer] 0004fb0000190000408d083065e062ff (Refresh Storage Layer)
      Operation: Adding a queued job operation

      Job Running Phase at 13:29 on Fri, Oct 26, 2012
      ----------------------------------------------
      Job Participants: [34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)]


      Actioner
      --------
      Starting operation 'Adding a queued job operation' on object '0004fb0000190000de6c2bd2cd7d2d79 (Refresh Storage Layer)'
      Completed operation 'Adding a queued job operation' completed with direction ==> LATER
      Starting operation 'Storage Server Discover Capabilities' on object '0004fb000009000022077528059c32a0 (ZFS-SA)'
      Setting Context to model only in job with id=1351220379118
      Setting Context to default in job with id=1351220379118
      Completed operation 'Storage Server Discover Capabilities' completed with direction ==> DONE
      Starting operation 'Storage Server Discover Access Groups' on object '0004fb000009000022077528059c32a0 (ZFS-SA)'
      Completed operation 'Storage Server Discover Access Groups' completed with direction ==> LATER
      Starting operation 'Adding a queued job operation' on object '0004fb0000190000408d083065e062ff (Refresh Storage Layer)'
      Completed operation 'Adding a queued job operation' completed with direction ==> LATER
      Starting operation 'Adding a queued job operation' on object '0004fb0000190000de6c2bd2cd7d2d79 (Refresh Storage Layer)'
      Completed operation 'Adding a queued job operation' completed with direction ==> LATER
      Starting operation 'Storage Array Discover Info' on object '0004fb000009000022077528059c32a0 (ZFS-SA)'
      Setting Context to model only in job with id=1351220379118
      Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_getInfo] failed for storage server [0004fb000009000022077528059c32a0] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
      at com.oracle.ovm.mgr.action.StoragePluginAction.discoverStorageArrayInformation(StoragePluginAction.java:285)
      at com.oracle.ovm.mgr.discover.ovm.StorageArrayInfoDiscoverHandler.query(StorageArrayInfoDiscoverHandler.java:27)
      at com.oracle.ovm.mgr.discover.ovm.StorageArrayInfoDiscoverHandler.query(StorageArrayInfoDiscoverHandler.java:16)
      at com.oracle.ovm.mgr.discover.ovm.DiscoverHandler.execute(DiscoverHandler.java:61)
      at com.oracle.ovm.mgr.discover.StorageServerDiscover.handleDiscover(StorageServerDiscover.java:77)
      at com.oracle.ovm.mgr.discover.StorageServerDiscover.discoverStorageServer(StorageServerDiscover.java:53)
      at com.oracle.ovm.mgr.op.physical.storage.StorageArrayDiscoverInfo.discoverStorageArrayInformation(StorageArrayDiscoverInfo.java:47)
      at com.oracle.ovm.mgr.op.physical.storage.StorageArrayDiscoverInfo.action(StorageArrayDiscoverInfo.java:38)
      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1012)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:329)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:289)
      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:826)
      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
      at com.oracle.ovm.mgr.api.physical.storage.IscsiStorageArrayProxy.executeCurrentJobOperationAction(Unknown Source)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:329)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:289)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
      at com.oracle.ovm.mgr.action.StoragePluginAction.discoverStorageArrayInformation(StoragePluginAction.java:281)
      ... 32 more
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
      ... 35 more


      FailedOperationCleanup
      ----------
      Starting failed operation 'Storage Array Discover Info' cleanup on object 'ZFS-SA'
      Complete rollback operation 'Storage Array Discover Info' completed with direction=ZFS-SA

      Rollbacker
      ----------
      Executing rollback operation 'Storage Array Discover Info' on object '0004fb000009000022077528059c32a0 (ZFS-SA)'
      Complete rollback operation 'Storage Array Discover Info' completed with direction=DONE
      Executing rollback operation 'Storage Server Discover Capabilities' on object '0004fb000009000022077528059c32a0 (ZFS-SA)'
      Complete rollback operation 'Storage Server Discover Capabilities' completed with direction=DONE

      Objects To Be Rolled Back
      -------------------------
      Object (CREATED): [ServerRefreshStorageLayer] 0004fb0000190000de6c2bd2cd7d2d79 (Refresh Storage Layer)
      Object (IN_USE): [IscsiStorageArray] 0004fb000009000022077528059c32a0 (ZFS-SA)
      Object (CREATED): [ServerRefreshStorageLayer] 0004fb0000190000408d083065e062ff (Refresh Storage Layer)


      Write Methods Invoked
      -------------------
      Class=InternalJobDbImpl vessel_id=73434 method=addTransactionIdentifier accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=refresh accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setName accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setFoundryContext accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=onPersistableCreate accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setLifecycleState accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setRollbackLifecycleState accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setTarget accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setStorageArray accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setSimpleName accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=addJobOperation accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setName accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setFoundryContext accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=onPersistableCreate accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setLifecycleState accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setRollbackLifecycleState accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setTarget accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setStorageArray accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setSimpleName accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=addJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setCompletedStep accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setAssociatedHandles accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setContext accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setMaxAccessGroupEntries accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setAbility accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setContext accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setCurrentJobOperationComplete accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setTuringMachineFlag accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setContext accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=73434 method=setFailedOperation accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73443 method=nextJobOperation accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=nextJobOperation accessLevel=6
      Class=ServerRefreshStorageLayerDbImpl vessel_id=73448 method=nextJobOperation accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=nextJobOperation accessLevel=6
      Class=IscsiStorageArrayDbImpl vessel_id=5757 method=nextJobOperation accessLevel=6
      Completed Step: ROLLBACK
      Job failed commit (internal) due to OVMAPI_B000E Storage plugin command [storage_plugin_getInfo] failed for storage server [0004fb000009000022077528059c32a0] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_getInfo] failed for storage server [0004fb000009000022077528059c32a0] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
      at com.oracle.ovm.mgr.action.StoragePluginAction.discoverStorageArrayInformation(StoragePluginAction.java:285)
      at com.oracle.ovm.mgr.discover.ovm.StorageArrayInfoDiscoverHandler.query(StorageArrayInfoDiscoverHandler.java:27)
      at com.oracle.ovm.mgr.discover.ovm.StorageArrayInfoDiscoverHandler.query(StorageArrayInfoDiscoverHandler.java:16)
      at com.oracle.ovm.mgr.discover.ovm.DiscoverHandler.execute(DiscoverHandler.java:61)
      at com.oracle.ovm.mgr.discover.StorageServerDiscover.handleDiscover(StorageServerDiscover.java:77)
      at com.oracle.ovm.mgr.discover.StorageServerDiscover.discoverStorageServer(StorageServerDiscover.java:53)
      at com.oracle.ovm.mgr.op.physical.storage.StorageArrayDiscoverInfo.discoverStorageArrayInformation(StorageArrayDiscoverInfo.java:47)
      at com.oracle.ovm.mgr.op.physical.storage.StorageArrayDiscoverInfo.action(StorageArrayDiscoverInfo.java:38)
      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1012)
      at sun.reflect.GeneratedMethodAccessor15397.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:329)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:289)
      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:826)
      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
      at com.oracle.ovm.mgr.api.physical.storage.IscsiStorageArrayProxy.executeCurrentJobOperationAction(Unknown Source)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
      at sun.reflect.GeneratedMethodAccessor1214.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:329)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:289)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
      at com.oracle.ovm.mgr.action.StoragePluginAction.discoverStorageArrayInformation(StoragePluginAction.java:281)
      ... 32 more
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_getInfo oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.LoginFailedEx:'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '
      Fri Oct 26 13:29:40 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
      ... 35 more


      ----------
      End of Job
      ----------

      Edited by: 942500 on Oct 25, 2012 8:02 PM

      Edited by: 942500 on Oct 25, 2012 8:03 PM
        • 1. Re: SAN fell over. Trying to recover my OVM pool.
          Dave Smulsky
          Have you verified all the user/passwords are setup the same, I see a lot of "'**** FAULT 47 : login denied\nWorkflow fault:: faultCode : 47faultString: login denied '" in your error messages suggesting otherwise.
          • 2. Re: SAN fell over. Trying to recover my OVM pool.
            user12273962
            If its just a server pool you're having issues with.... then just recreate it and move you servers into that pool. Are you talking about a storage repos?

            I'm curious about the ZFS appliance issue? I run one myself. A 7420. Are you running the virtual box appliance? or a appliance itself?
            • 3. Re: SAN fell over. Trying to recover my OVM pool.
              945503
              Running the ZFS-SA as a VM. Hugely bad idea, I know... so I'm not too surprised that it died. VirtualBox somehow managed to create a file in the VMs "snapshot" directory that filled the host server's disk and caused the SAN VM to crash.
              Funny really, because I never created any snapshots on that VM (why would I?)

              It has been suggested to blow away the pool and recreate it and present the disks back to it. That would be great if I wasn't in a catch-22 lockup with the old SAN recorded in the OVM manager. I cannot unpresent the "old" SAN record (it cannot see the old SAN to unpresent it), and I cannot re-add the "new" SAN (because it clashes with the old record on the same IP address).

              What I'm trying to find out is how to recover from such a situation. If I have a SAN in future where the OS disk for the SAN dies... how could I rebuild it and present it to the system without too much downtime.

              This has been a constant nagging issue for me with OVM. Sure, it's never good to have a SAN ripped out from under any system.... in vmware, for example, you can recover from such a situation.
              In the case of OVM it typically results in a cascading set of problems, each a catch-22, where the only way out is to blow everything away, rebuild the entire pool and recover everything from backup (if you have any).
              Needless to say... that really sucks.

              Edited by: 942500 on Oct 28, 2012 4:02 PM
              • 4. Re: SAN fell over. Trying to recover my OVM pool.
                user12273962
                It probably is a bad idea to run the Virtual box ZFS appliance. There has been a lot of updates to the ZFS appliance software within the last year. I don't know where the VB ZFS is at as far as updates.

                I've never had much of a problem deleting a SAN entry. It can be difficult at times to tear away everything that is using the SAN in order to remove the old entry. I would recommend just creating a OEL5 or OEL 6 NFS server. Even use virtual box to do it. Then use that server for central storage. Or if you like to use ISCSI... then try open filler. http://www.openfiler.com or you can use the native features of OEL5 or OEL6. It can be aggravating to use Iscsi in Linux if you've never done it before. Open filler works great. Never tried their virtual appliance on virtual box.

                When VM runs out of space. (it happened to me once) you can usually do a "ocfs2 file check" from the mount points and everything will be okay. At least it was with me. Little aggravating but that is to be expected.