1 Reply Latest reply: Apr 15, 2012 5:29 AM by 892072 RSS

    Vm Servers failing to start - Urgent HELP

    892072
      Hi

      I have suddenly found I cannot start any of my vm`s on our Oracle 3.0.3 vm server setup. If I stop any running vm`s I then get the following error on trying to start them.
      Its the weekend and im trying to do some work while the office is emtpy and am now totally stuck and in trouble.

      Also I do have a CSI (ordered 3 days ago) but aparently is not active (just called help line) and they cannot find any record of it. Something about takes 36 hours and I need to receive a welcome letter? As its the weekend (sun) I will have to wait until next week.

      We currently have two vm hosts running for last few months without any problems. I did add on Friday a third host server but cannot really see how this could have affected it.

      Error on starting vm`s follows

      Any help/response would be nice

      Nick Colyer


      Job Construction Phase
      ----------------------
      begin()
      Appended operation 'Virtual Machine Start' to object '0004fb00000600004582721d06a8ab47 (scripts.ex)'.
      Appended operation 'Network Port Configure' to object 'network.EthernetPort (2) in 53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc with MAC 00:25:90:38:dc:bf (network.EthernetPort (2) in ovm2.ex)'.
      Appended operation 'VM Set HA' to object '0004fb00000600004582721d06a8ab47 (scripts.ex)'.
      commit()
      Completed Step: COMMIT

      Objects and Operations
      ----------------------
      Object (IN_USE): [EthernetPort] network.EthernetPort (2) in 53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc with MAC 00:25:90:38:dc:bf (network.EthernetPort (2) in ovm2.ex)
      Operation: Network Port Configure
      Object (IN_USE): [VmDiskMapping] 0004fb0000130000f945246fe9077021
      Object (IN_USE): [VirtualMachine] 0004fb00000600004582721d06a8ab47 (scripts.ex)
      Operation: Virtual Machine Start
      Operation: VM Set HA
      Object (IN_USE): [VirtualDisk] scripts.img
      Object (IN_USE): [Vnic] 0004fb0000070000dc0557c1dac9a1b8 (00:21:f6:33:00:10)

      Job Running Phase at 10:12 on Sun, Apr 15, 2012
      ----------------------------------------------
      Job Participants: [53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc (ovm2.ex)]


      Actioner
      --------
      Starting operation 'Network Port Configure' on object 'network.EthernetPort (2) in 53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc with MAC 00:25:90:38:dc:bf (network.EthernetPort (2) in ovm2.ex)'
      Completed operation 'Network Port Configure' completed with direction ==> DONE
      Starting operation 'Virtual Machine Start' on object '0004fb00000600004582721d06a8ab47 (scripts.ex)'
      Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm2.ex failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.11.8.120:8899/api/1 start_vm 0004fb000003000026feea96455f02e2 0004fb00000600004582721d06a8ab47, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command ['xm', 'create', '/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg'] failed (1): stderr: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
      , stdout: Using config file "/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg".

      Sun Apr 15 10:15:29 BST 2012
      Sun Apr 15 10:15:29 BST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:415)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:362)
      at com.oracle.ovm.mgr.action.VirtualMachineAction.sendStartVmCommand(VirtualMachineAction.java:121)
      at com.oracle.ovm.mgr.op.virtual.VirtualMachineStart.asyncAction(VirtualMachineStart.java:65)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:573)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:564)
      at com.oracle.ovm.mgr.op.virtual.VirtualMachineStart.action(VirtualMachineStart.java:52)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:751)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.11.8.120:8899/api/1 start_vm 0004fb000003000026feea96455f02e2 0004fb00000600004582721d06a8ab47, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command ['xm', 'create', '/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg'] failed (1): stderr: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
      , stdout: Using config file "/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg".

      Sun Apr 15 10:15:29 BST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:753)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:471)
      ... 23 more


      FailedOperationCleanup
      ----------
      Starting failed operation 'Virtual Machine Start' cleanup on object 'scripts.ex'
      Complete rollback operation 'Virtual Machine Start' completed with direction=scripts.ex

      Rollbacker
      ----------
      Executing rollback operation 'Network Port Configure' on object 'network.EthernetPort (2) in 53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc with MAC 00:25:90:38:dc:bf (network.EthernetPort (2) in ovm2.ex)'
      Complete rollback operation 'Network Port Configure' completed with direction=DONE

      Objects To Be Rolled Back
      -------------------------
      Object (IN_USE): [EthernetPort] network.EthernetPort (2) in 53:4d:43:49:00:02:38:90:25:00:38:90:25:00:bf:dc with MAC 00:25:90:38:dc:bf (network.EthernetPort (2) in ovm2.ex)
      Object (IN_USE): [VmDiskMapping] 0004fb0000130000f945246fe9077021
      Object (IN_USE): [VirtualMachine] 0004fb00000600004582721d06a8ab47 (scripts.ex)
      Object (IN_USE): [VirtualDisk] scripts.img
      Object (IN_USE): [Vnic] 0004fb0000070000dc0557c1dac9a1b8 (00:21:f6:33:00:10)


      Write Methods Invoked
      -------------------
      Class=InternalJobDbImpl vessel_id=34470 method=addTransactionIdentifier accessLevel=6
      Class=VirtualMachineDbImpl vessel_id=2790 method=start accessLevel=6
      Class=VnicDbImpl vessel_id=1183 method=lock accessLevel=6
      Class=VirtualDiskDbImpl vessel_id=2775 method=lock accessLevel=6
      Class=VnicDbImpl vessel_id=1183 method=lock accessLevel=6
      Class=VmDiskMappingDbImpl vessel_id=2807 method=lock accessLevel=6
      Class=VirtualDiskDbImpl vessel_id=2775 method=lock accessLevel=6
      Class=VirtualMachineStartingEventDbImpl vessel_id=34479 method=notify accessLevel=6
      Class=VirtualMachineStartingEventDbImpl vessel_id=34479 method=setDescription accessLevel=6
      Class=EthernetPortDbImpl vessel_id=4943 method=addJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=34470 method=setCompletedStep accessLevel=6
      Class=InternalJobDbImpl vessel_id=34470 method=setAssociatedHandles accessLevel=6
      Class=InternalJobDbImpl vessel_id=34470 method=setAsset accessLevel=6
      Class=EthernetPortDbImpl vessel_id=4943 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=34470 method=setFailedOperation accessLevel=6
      Class=EthernetPortDbImpl vessel_id=4943 method=nextJobOperation accessLevel=6
      Class=VmDiskMappingDbImpl vessel_id=2807 method=nextJobOperation accessLevel=6
      Class=VirtualMachineDbImpl vessel_id=2790 method=nextJobOperation accessLevel=6
      Class=VirtualDiskDbImpl vessel_id=2775 method=nextJobOperation accessLevel=6
      Class=VnicDbImpl vessel_id=1183 method=nextJobOperation accessLevel=6
      Class=EthernetPortDbImpl vessel_id=4943 method=nextJobOperation accessLevel=6
      Completed Step: ROLLBACK
      Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: dispatch to server: ovm2.ex failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.11.8.120:8899/api/1 start_vm 0004fb000003000026feea96455f02e2 0004fb00000600004582721d06a8ab47, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command ['xm', 'create', '/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg'] failed (1): stderr: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
      , stdout: Using config file "/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg".

      Sun Apr 15 10:15:29 BST 2012
      Sun Apr 15 10:15:29 BST 2012
      com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm2.ex failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.11.8.120:8899/api/1 start_vm 0004fb000003000026feea96455f02e2 0004fb00000600004582721d06a8ab47, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command ['xm', 'create', '/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg'] failed (1): stderr: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
      , stdout: Using config file "/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg".

      Sun Apr 15 10:15:29 BST 2012
      Sun Apr 15 10:15:29 BST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:415)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:362)
      at com.oracle.ovm.mgr.action.VirtualMachineAction.sendStartVmCommand(VirtualMachineAction.java:121)
      at com.oracle.ovm.mgr.op.virtual.VirtualMachineStart.asyncAction(VirtualMachineStart.java:65)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:573)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:564)
      at com.oracle.ovm.mgr.op.virtual.VirtualMachineStart.action(VirtualMachineStart.java:52)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
      at sun.reflect.GeneratedMethodAccessor886.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:751)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.11.8.120:8899/api/1 start_vm 0004fb000003000026feea96455f02e2 0004fb00000600004582721d06a8ab47, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command ['xm', 'create', '/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg'] failed (1): stderr: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
      , stdout: Using config file "/OVS/Repositories/0004fb000003000026feea96455f02e2/VirtualMachines/0004fb00000600004582721d06a8ab47/vm.cfg".

      Sun Apr 15 10:15:29 BST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:753)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:471)
      ... 23 more


      ----------
      End of Job
      ----------
        • 1. Re: Vm Servers failing to start - Urgent HELP
          892072
          Looks like I have got around this problem.

          I rebooted the 3rd (new) host server and was able to live migrate servers over to it. It would also allow me to start stopped servers on it as well.
          Just had to live migrate all the servers from the other two host servers and reboot the host machines.

          Once done everything seems to work and I can stop/start servers on any of the host servers.

          Nick