1 Reply Latest reply: Feb 19, 2014 9:15 AM by Terry Phelps RSS

    Cannot remove a physical disk from a VM

    Terry Phelps

      I'm running Oracle VM 3.2.4. I have added and removed physical disks from VMs many times without any problem. Today I cannot remove a physical disk.

       

      No doubt it is related to what I did just before this: I resized the LUN. I do NOT have any Oracle VM storage plugins. My SAN is an IBM XIV, if it matters to you. Here's what I think I did:

      Unmounted the filesystem in the VM.

      Removed the partition on the LUN, since I was happy to completely wipe out the disk.

      From the SAN, I resized the LUN from 16 to 32 GB.

      I did a 'rescan physical disks' on each Oracle VM server, in the fond hope that he would see the new LUN size, and things would be okay. (But I've NEVER had the 'rescan physical disks' from OVMM do anything. So, I did what I always do when I add or remove a LUN...)

      On each Oracle VM server, I did a "rescan-scsi-bus.sh". This has always caused the OVMM to update his list of physical disks. But this time, nothing happened.

       

      Okay, I thought, I'll remove the physical disk from the VM, remove it from OVMM, un-present it from the SAN, and add it all back again.

      I edit the VM, and removed the physical disk, like I've done many times, and hit OK. It thinks for many seconds, and this FAILs. I'll paste in the details, after I'm finished writing here.

       

      So, I can't remove the LUN from the VM, and I can't remove the LUN from the "FibreChannel Volume Group", because it's mapped into a VM.

      I have restarted OVMM, but that didn't do anything.

       

      Any ideas??

       

      Okay, here are the error details:

       

      Job Construction Phase

      ----------------------

      Job ID: 1392820419550

       

      begin()

      Appended operation 'Virtual Machine Cfg File Configure' to object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'.

      commit()

      Completed Step: COMMIT

       

      Objects and Operations

      ----------------------

      Object (IN_USE): [VirtualMachine] 0004fb0000060000d69fd57f4fdd0830 (jidldb01)

      Object (IN_USE): [StorageElement] 0004fb0000180000290d6bf2dac7f445 (jidldb01_tldev)

      Object (IN_USE): [Vnic] 0004fb0000070000105a685f23193f8f (00:21:f6:00:00:00)

      Object (DELETING): [VmDiskMapping] 0004fb0000130000449523548475f2e0

      Object (IN_USE): [VirtualMachineCfgFile] cfgFile_0004fb0000060000d69fd57f4fdd0830

      Operation: Virtual Machine Cfg File Configure

       

      Job Running Phase at 2014-02-19 09:33:39,550

      ----------------------------------------------

      Job Participants: [26:bc:94:d8:86:91:e1:11:81:c8:e4:1f:13:eb:d8:d2 (oravm1.acbl.net)]

       

       

      Actioner

      --------

      09:33:40,092: Starting operation 'Virtual Machine Cfg File Configure' on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'

      Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.

      stdout: Usage: xm block-detach [-f|--force]

       

      Destroy a domain's virtual block device.

      [Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]

      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:512)

      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:449)

      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:383)

      at com.oracle.ovm.mgr.action.VirtualMachineCfgFileAction.configureVirtualMachineCfgFile(VirtualMachineCfgFileAction.java:152)

      at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.configureVirtualMachineCfgFile(VirtualMachineCfgFileConfigure.java:82)

      at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.action(VirtualMachineCfgFileConfigure.java:55)

      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)

      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:865)

      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)

      at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)

      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)

      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)

      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1357)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)

      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)

      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)

      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)

      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)

      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)

      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)

      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)

      at java.lang.Thread.run(Thread.java:662)

      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.

      stdout: Usage: xm block-detach [-f|--force]

       

      Destroy a domain's virtual block device.

      [Wed Feb 19 09:35:20 EST 2014]

      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:803)

      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:508)

      ... 32 more

       

       

      FailedOperationCleanup

      ----------

      Starting failed operation 'Virtual Machine Cfg File Configure' cleanup on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'

      Complete rollback operation 'Virtual Machine Cfg File Configure' cleanup on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'

       

      Rollbacker

      ----------

      09:35:20,799: Starting rollbacker...

      Executing rollback operation 'Virtual Machine Cfg File Configure' on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'

      Complete rollback operation 'Virtual Machine Cfg File Configure' completed with direction=DONE

      09:35:20,817: Rollbacker completed...

       

      Objects To Be Rolled Back

      -------------------------

      Object (IN_USE): [StorageElement] 0004fb0000180000290d6bf2dac7f445 (jidldb01_tldev)

      Object (IN_USE): [VirtualMachine] 0004fb0000060000d69fd57f4fdd0830 (jidldb01)

      Object (IN_USE): [Vnic] 0004fb0000070000105a685f23193f8f (00:21:f6:00:00:00)

      Object (DELETING): [VmDiskMapping] 0004fb0000130000449523548475f2e0

      Object (IN_USE): [VirtualMachineCfgFile] cfgFile_0004fb0000060000d69fd57f4fdd0830

       

       

      Write Methods Invoked

      -------------------

      09:33:39,762 class="InternalJobDbImpl" vessel_id=204501 method=addTransactionIdentifier accessLevel=6 owningTx=1392820419762

      09:33:39,766 class="VnicDbImpl" vessel_id=1656 method=setEthernetNetwork accessLevel=6 owningTx=1392820419762

      09:33:39,767 class="VirtualMachineDbImpl" vessel_id=98707 method=recordCfgChange accessLevel=6 owningTx=1392820419762

      09:33:39,767 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762

      09:33:39,768 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762

      09:33:39,788 class="VirtualMachineDbImpl" vessel_id=98707 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762

      09:33:39,789 class="StorageElementDbImpl" vessel_id=203407 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762

      09:33:39,789 class="VmDiskMappingDbImpl" vessel_id=203762 method=setVirtualMachineDisk accessLevel=6 owningTx=1392820419762

      09:33:39,790 class="VirtualMachineDbImpl" vessel_id=98707 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762

      09:33:39,790 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762

      09:33:39,790 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762

      09:33:39,791 class="VmDiskMappingDbImpl" vessel_id=203762 method=setVirtualMachine accessLevel=6 owningTx=1392820419762

      09:33:39,808 class="ModelManagerDbImpl" vessel_id=9 method=deleteObject accessLevel=5 owningTx=-1

      09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=setLifecycleState accessLevel=6 owningTx=1392820419762

      09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=setRollbackLifecycleState accessLevel=6 owningTx=1392820419762

      09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=onPersistableClean accessLevel=6 owningTx=1392820419762

      09:33:39,809 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762

      09:33:39,809 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762

      09:33:39,940 class="InternalJobDbImpl" vessel_id=204501 method=setCompletedStep accessLevel=6 owningTx=1392820419762

      09:33:39,945 class="InternalJobDbImpl" vessel_id=204501 method=setAssociatedHandles accessLevel=6 owningTx=1392820419762

      09:33:40,139 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setChecksum accessLevel=6 owningTx=1392820419762

      09:35:20,654 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,654 class="InternalJobDbImpl" vessel_id=204501 method=setFailedOperation accessLevel=6 owningTx=1392820419762

      09:35:20,803 class="VirtualMachineDbImpl" vessel_id=98707 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,803 class="StorageElementDbImpl" vessel_id=203407 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,804 class="VnicDbImpl" vessel_id=1656 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,804 class="VmDiskMappingDbImpl" vessel_id=203762 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,804 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      09:35:20,817 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762

      Completed Step: ROLLBACK

      Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.

      stdout: Usage: xm block-detach [-f|--force]

       

      Destroy a domain's virtual block device.

      [Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]

      com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.

      stdout: Usage: xm block-detach [-f|--force]

       

      Destroy a domain's virtual block device.

      [Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]

      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:512)

      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:449)

      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:383)

      at com.oracle.ovm.mgr.action.VirtualMachineCfgFileAction.configureVirtualMachineCfgFile(VirtualMachineCfgFileAction.java:152)

      at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.configureVirtualMachineCfgFile(VirtualMachineCfgFileConfigure.java:82)

      at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.action(VirtualMachineCfgFileConfigure.java:55)

      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)

      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:865)

      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)

      at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)

      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)

      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)

      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1357)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)

      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)

      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)

      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)

      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)

      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)

      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)

      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)

      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)

      at java.lang.Thread.run(Thread.java:662)

      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.

      stdout: Usage: xm block-detach [-f|--force]

       

      Destroy a domain's virtual block device.

      [Wed Feb 19 09:35:20 EST 2014]

      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:803)

      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:508)

      ... 32 more

       

       

      ----------

      End of Job

      ----------

        • 1. Re: Cannot remove a physical disk from a VM
          Terry Phelps

          NEVER MIND.

           

          I found the problem. OVMM wouldn't remove the LUN fromt he VM because it was still in use. I had unmounted the filesystem on the disk, BUT I had forgotten that there was a logical volume, and volume group under that filesystem. Those were still sort of there, and the disk was in use. I did a "dmsetup remove" on the volume group, and all is well.