This discussion is archived
11 Replies Latest reply: Nov 15, 2012 9:11 AM by Roddy Rodstein RSS

Oracle VM Manager  3.1.1.478 starts the same VM twice!

Roddy Rodstein Explorer
Currently Being Moderated
Greetings,

I had to share this Oracle VM 3.1.1.399 & with 3.1.1.478 experience with y’all.

We have a two server pool with HPG6 Blades, EMC/FC/SAN storage, 1 12G Poolfs LUN, and 2 1TB LUNs for the VMs with a toal of 5 tagged VLANs, i.e. eth0/eth1 bond0 mode4 (3 vlans), eth2/eth3 bond1 mode 4 (2 vlans). Oracle VM Manager is installed on a VM OL58 has the latest patch .399 and both hosts have been patched via a local repo last week.

On two occasions Oracle VM Manager has started the same VM (OL5U8 PVM w UEK) at the same time on two hosts! As you could imagine having the same VM running twice causes some intresting issues :-)

We will be opening an SR lickety split, although in the mean time I wanted to see if you have ever seen this one, if yes, whats the root cause and fix?

UPDATE:
FYI, we can recreate by restarting any VM using Oracle VM Manager. The VM is restarted on the origianl host and also started the second host. We have a two server pool.

[root@ovs715 ~]
[2012-09-26 10:52:35 7258] DEBUG (service:77) reboot_vm: ('0004fb0000030000342c576139f91922', '0004fb00000600002eeb36088b01069a', True)
[2012-09-26 10:53:30 5991] DEBUG (ha:54) Trying to start VM 0004fb00000600002eeb36088b01069a
[2012-09-26 10:53:34 5982] INFO (notification:193) Sending notification: {DOMAIN} 0004fb00000600002eeb36088b01069a {SSLTTY} 7901
[2012-09-26 10:53:35 5982] INFO (notification:193) Sending notification: {DOMAIN} 0004fb00000600002eeb36088b01069a {SSLTTY} 7902
[2012-09-26 10:53:35 5991] DEBUG (ha:63) VM 0004fb00000600002eeb36088b01069a started on 34.36.204.72
[2012-09-26 10:53:35 5982] INFO (notification:193) Sending notification: {DOMAIN} 0004fb00000600002eeb36088b01069a {SSLVNC} 6900
[root@ovs716 ~]
[2012-09-26 10:53:30 9836] DEBUG (service:77) start_vm: ('0004fb0000030000342c576139f91922', '0004fb00000600002eeb36088b01069a')
[2012-09-26 10:53:35 5903] INFO (notification:193) Sending notification: {DOMAIN} 0004fb00000600002eeb36088b01069a {SSLVNC} 6903
[2012-09-26 10:53:36 5903] INFO (notification:193) Sending notification: {DOMAIN} 0004fb00000600002eeb36088b01069a {SSLTTY} 7909


Respectfully,

Edited by: Roddy Rodstein on Sep 26, 2012 9:08 AM

Edited by: Roddy Rodstein on Nov 15, 2012 9:11 AM
  • 1. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    user12273962 Pro
    Currently Being Moderated
    If you're using powerpath or not using it.... it might be the issue. The last time I checked.. Powerpath wasn't released/supported for 3.1.1

    Edited by: user12273962 on Sep 25, 2012 11:26 AM
  • 2. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Thanks for your reply! All defaults, i.e. dm-multipath.
  • 3. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    user157995 Explorer
    Currently Being Moderated
    Are you sure it actually started it twice, or just displayed it twice in the manager?

    Ex) output of "xm list" on the dom0 would be helpful to detect this.
  • 4. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    user157995 Explorer
    Currently Being Moderated
    We had this issue early on (3.0.2 I think) where we didnt setup DNS for the OVM hostnames/ips and the manager was trying to query the destination node during live migration via its unresolvable hostname, which ended up causing the live migration to somehow "crash" which caused 2 identical VMs to start (which resulted in a 400GB restore of a Oracle 10G database... :( )
  • 5. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Thank you for your assistance!

    Oracle VM Manager showed the offending VM only once in the GUI although it was indeed running on each of our two hosts at the same time!

    As requested, lots O data, names have been changed to protect the innocent :-) :
    [root@ovs716 ~]# find /OVS/Repositories -type f -name vm.cfg -exec grep -iH simple_name {} \; | grep ovm724v
    /OVS/Repositories/0004fb0000030000342c576139f91922/VirtualMachines/0004fb00000600002466c345c6f506d1/vm.cfg:OVM_simple_name = 'ovm724v'

    [root@ovs715 ~]# find /OVS/Repositories -type f -name vm.cfg -exec grep -iH simple_name {} \; | grep ovm724v
    /OVS/Repositories/0004fb0000030000342c576139f91922/VirtualMachines/0004fb00000600002466c345c6f506d1/vm.cfg:OVM_simple_name = 'ovm724v'

    xm list was taken at the same time from both hosts, 0004fb00000600002466c345c6f506d1 is our huckleberry:
    [root@ovs716 ~]# xm list
    Name ID Mem VCPUs State Time(s)
    0004fb00000600002466c345c6f506d1 28 24576 4 -b---- 39.4
    0004fb0000060000698cc38852aa7861 13 16000 16 -b---- 95527.5
    0004fb00000600008eeb92db5c758f64 14 24000 16 -b---- 97225.9
    Domain-0 0 3132 24 r----- 258697.2

    [root@ovs715 ~]# xm list
    Name ID Mem VCPUs State Time(s)
    0004fb00000600002466c345c6f506d1 26 24576 4 -b---- 3230.3
    0004fb00000600002eeb36088b01069a 25 8192 2 -b---- 1240.5
    0004fb0000060000802736e145880e03 16 8000 8 -b---- 1331.1
    Domain-0 0 3132 24 r----- 507903.6

    [root@ovs716 ~]# xm list -l 28
    (domain
    (domid 28)
    (cpu_weight 27500)
    (cpu_cap 0)
    (on_crash restart)
    (uuid 0004fb00-0006-0000-2466-c345c6f506d1)
    (bootloader_args -q)
    (pool_name Pool-0)
    (vcpus 4)
    (name 0004fb00000600002466c345c6f506d1)
    (on_poweroff destroy)
    (on_reboot restart)
    (cpus (() () () ()))
    (description '')
    (bootloader /usr/bin/pygrub)
    (maxmem 24576)
    (memory 24576)
    (shadow_memory 0)
    (features '')
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (start_time 1348587674.79)
    (cpu_time 39.435029815)
    (online_vcpus 4)
    (image
    (linux
    (kernel '')
    (expose_host_uuid 0)
    (superpages 0)
    (tsc_mode 0)
    (videoram 4)
    (pci ())
    (nomigrate 0)
    (device_model /usr/lib/xen/bin/qemu-dm)
    (notes
    (HV_START_LOW 18446603336221196288)
    (FEATURES '!writable_page_tables|pae_pgdir_above_4gb')
    (VIRT_BASE 18446744071562067968)
    (GUEST_VERSION 2.6)
    (PADDR_OFFSET 0)
    (GUEST_OS linux)
    (HYPERCALL_PAGE 18446744071578882048)
    (LOADER generic)
    (SUSPEND_CANCEL 1)
    (PAE_MODE yes)
    (ENTRY 18446744071590892032)
    (XEN_VERSION xen-3.0)
    )
    )
    )
    (status 2)
    (state -b----)
    (store_mfn 22742075)
    (console_mfn 22742074)
    (device
    (vif
    (bridge 0004fb0010a0c50)
    (mac 00:21:f6:00:00:42)
    (script /etc/xen/scripts/vif-bridge)
    (uuid be4676ca-c181-9b9f-a671-08cd54aa2120)
    (backend 0)
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 03e5afd5-4902-3a4e-fbd2-e023da35f95b)
    (bootable 1)
    (dev xvda:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb0000120000ecef7e315b42e25b.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 953c488c-5efa-4357-21f3-67fc731442f5)
    (bootable 0)
    (dev xvdb:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb0000120000cbb8d2c19c68f88d.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid e135e582-081f-dc6a-3db0-57d6c48783ec)
    (bootable 0)
    (dev xvdc:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb000012000008651cb7d81631c6.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 6e36982a-f4d8-d8e6-b29d-cd857a52fae5)
    (bootable 0)
    (dev xvdd:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb000012000060776ea3de034c44.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 0d885d06-387b-6dac-741f-64832dc89426)
    (bootable 0)
    (dev xvde:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb00001200008ba3f909e2dfd0e6.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device (vkbd (backend 0)))
    (device
    (vfb
    (vncunused 1)
    (vnc 1)
    (uuid 90e966e1-44c5-10e2-a839-fa1f10f7c225)
    (vnclisten 127.0.0.1)
    (keymap en-us)
    (location 127.0.0.1:5902)
    (xauthority //.Xauthority)
    )
    )
    (device
    (console
    (protocol vt100)
    (location 2)
    (uuid d375e894-7621-741c-ffe9-94de0134f805)
    )
    )
    )

    [root@ovs715 ~]# xm list
    Name ID Mem VCPUs State Time(s)
    0004fb00000600002466c345c6f506d1 26 24576 4 -b---- 3230.3
    0004fb00000600002eeb36088b01069a 25 8192 2 -b---- 1240.5
    0004fb0000060000802736e145880e03 16 8000 8 -b---- 1331.1
    Domain-0 0 3132 24 r----- 507903.6
    [root@ovs715 ~]# xm list -l 26
    (domain
    (domid 26)
    (cpu_weight 27500)
    (cpu_cap 0)
    (on_crash restart)
    (uuid 0004fb00-0006-0000-2466-c345c6f506d1)
    (bootloader_args -q)
    (pool_name Pool-0)
    (vcpus 4)
    (name 0004fb00000600002466c345c6f506d1)
    (on_poweroff destroy)
    (on_reboot restart)
    (cpus (() () () ()))
    (description '')
    (bootloader /usr/bin/pygrub)
    (maxmem 24576)
    (memory 24576)
    (shadow_memory 0)
    (features '')
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (start_time 1348519079.52)
    (cpu_time 3230.94067026)
    (online_vcpus 4)
    (image
    (linux
    (kernel '')
    (expose_host_uuid 0)
    (superpages 0)
    (tsc_mode 0)
    (videoram 4)
    (pci ())
    (nomigrate 0)
    (device_model /usr/lib/xen/bin/qemu-dm)
    (notes
    (HV_START_LOW 18446603336221196288)
    (FEATURES '!writable_page_tables|pae_pgdir_above_4gb')
    (VIRT_BASE 18446744071562067968)
    (GUEST_VERSION 2.6)
    (PADDR_OFFSET 0)
    (GUEST_OS linux)
    (HYPERCALL_PAGE 18446744071578882048)
    (LOADER generic)
    (SUSPEND_CANCEL 1)
    (PAE_MODE yes)
    (ENTRY 18446744071590892032)
    (XEN_VERSION xen-3.0)
    )
    )
    )
    (status 2)
    (state -b----)
    (store_mfn 18687119)
    (console_mfn 18687118)
    (device
    (vif
    (bridge 0004fb0010a0c50)
    (mac 00:21:f6:00:00:42)
    (script /etc/xen/scripts/vif-bridge)
    (uuid c74e2321-2e76-96ea-6a14-3a85be7fa3bb)
    (backend 0)
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 8acc20c1-5d07-5f38-aa68-899f6ceaecec)
    (bootable 1)
    (dev xvda:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb0000120000ecef7e315b42e25b.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 68bab31e-bd53-3bfa-4acb-8a2d13ceeb26)
    (bootable 0)
    (dev xvdb:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb0000120000cbb8d2c19c68f88d.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 0bbee7d7-8833-1d2e-6166-780b8945cece)
    (bootable 0)
    (dev xvdc:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb000012000008651cb7d81631c6.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid b38706b8-9462-1b62-d84a-dfb215b07992)
    (bootable 0)
    (dev xvdd:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb000012000060776ea3de034c44.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device
    (vbd
    (protocol x86_64-abi)
    (uuid 4decf667-77db-5942-e756-a733e875a7b8)
    (bootable 0)
    (dev xvde:disk)
    (uname
    file:/OVS/Repositories/0004fb0000030000342c576139f91922/VirtualDisks/0004fb00001200008ba3f909e2dfd0e6.img
    )
    (mode w)
    (backend 0)
    (VDI '')
    )
    )
    (device (vkbd (backend 0)))
    (device
    (vfb
    (vncunused 1)
    (vnc 1)
    (uuid 098aeeb1-da99-d34f-def2-c3f6b86fec1e)
    (vnclisten 127.0.0.1)
    (keymap en-us)
    (location 127.0.0.1:5902)
    (xauthority //.Xauthority)
    )
    )
    (device
    (console
    (protocol vt100)
    (location 2)
    (uuid 35872c6c-6ff8-47cf-31bc-a78bf9b13f74)
    )
    )
    )
  • 6. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Wow, thats CRAZY! I double checked, name resolution is a-o-k with dns forward and reverse lookups. Thats would have been a quick fix!

    Thank you for your help!
  • 7. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    user12273962 Pro
    Currently Being Moderated
    Is your repo on FC?

    I've never personally tried to run native linux mulitpathing on a EMC storage environment. It would interesting to see if single pathing the storage will fix it.
  • 8. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Thanks for your support! Yes we are using FC/SAN.

    We will keep the "single pathing the storage" use case in mind if O support cant assist. So far no love :-)
  • 9. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Sev 1 SR, 3+ days and not a peep from O support!

    We did isolate and resolve the issue!

    We uninstalled Manager and dropped the ovs/EE DB user, then UUID restored Manager and the problem was gone.

    An Oracle VM Manager bug, a corrupt Manager DB?

    Edited by: Roddy Rodstein on Oct 1, 2012 6:01 AM
  • 10. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    user12273962 Pro
    Currently Being Moderated
    I feel your pain. I've had a Sev2 open on a issue that has spent the lasts 18 days in "review update". The only way to move SR like this along is to call and hound someone to death. Maybe get to speak to a duty manager. Even that doesn't help sometimes. I haven't had to reinstall the VM Manager yet but I can see it coming.
  • 11. Re: Oracle VM Manager 3.1.1.399 starts the same VM twice!
    Roddy Rodstein Explorer
    Currently Being Moderated
    Greetings,

    FYI, this bug is a BACK with a vengence! VMs start twice, when you init0 or stop VMs in Manager with HA enabled they a) restart on thier own where they please and/or start twice and corrupt thier disks!

    Diable HA on all VMs looks like a short term solution, until we get a patch.

    OVM Versions:
    Oracle VM Manager patch ID 14227416
    Oracle VM Server with the latest patches, i.e. 11/15/12.

    O support is on the case :-)

    Will check back with updates a bit later.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points