0 Replies Latest reply on Jan 28, 2014 5:53 AM by a8068bb7-eec6-465a-9a83-f62ee1847f1f

    OVM - 3.2.2 - VM crashes &  reboots while running heavy TCP/UDP  traffic

    a8068bb7-eec6-465a-9a83-f62ee1847f1f

      I am seeing an issue while RHEL VM crashes and reboots while starting heavy TCP/UDP traffic .

       

      However I dint find any 10G CNA driver  related error messages at server in logs . Pl provide comments if you have encountered any similar situation.

       

      Test Network adapter – 10 G NIC CNA,   OS – OVM 3.2.2,  VM -  RHEL_6.4, 

       

      Messages seen At VM ..

       

      Jan 27 08:58:18 RHEL-LEX kernel: __ratelimit: 109 callbacks suppressed

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:18 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:19 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:19 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:19 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:19 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:23 RHEL-LEX kernel: __ratelimit: 153 callbacks suppressed

      Jan 27 08:58:23 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:23 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:23 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: Invalid extra type: 48

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: Invalid extra type: 55

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: Missing extra info

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: Need more frags

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:24 RHEL-LEX kernel: net eth0: rx->offset: 0, size: 4294967295

      Jan 27 08:58:25 RHEL-LEX kernel: BUG: scheduling while atomic: swapper/0/0x10000100

      Jan 27 08:58:25 RHEL-LEX kernel: Modules linked in: bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc cnic uio fcoe libfcoe sg libfc 8021q scsi_transport_fc scsi_tgt garp stp llc ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput ppdev parport_pc parport microcode xen_netfront i2c_piix4 i2c_core ext4 jbd2 mbcache xen_blkfront pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]

      Jan 27 08:58:25 RHEL-LEX kernel: CPU 0

      Jan 27 08:58:25 RHEL-LEX kernel: Modules linked in: bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc cnic uio fcoe libfcoe sg libfc 8021q scsi_transport_fc scsi_tgt garp stp llc ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput ppdev parport_pc parport microcode xen_netfront i2c_piix4 i2c_core ext4 jbd2 mbcache xen_blkfront pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]

      Jan 27 08:58:25 RHEL-LEX kernel:

      Jan 27 08:58:25 RHEL-LEX kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 Xen HVM domU

      Jan 27 08:58:25 RHEL-LEX kernel: RIP: 0010:[<ffffffff810ad7f6>]  [<ffffffff810ad7f6>] tick_nohz_stop_sched_tick+0x296/0x3c0

      Jan 27 08:58:25 RHEL-LEX kernel: RSP: 0018:ffffffff81a01e68  EFLAGS: 00000282

      Jan 27 08:58:25 RHEL-LEX kernel: RAX: 00000000022330d1 RBX: ffffffff81a01ec8 RCX: 0000000000000020

      Jan 27 08:58:25 RHEL-LEX kernel: RDX: 000000010031911c RSI: 0000000000000086 RDI: 0000000000000282

      Jan 27 08:58:25 RHEL-LEX kernel: RBP: ffffffff8100c42e R08: ffff88002820e100 R09: ffff8800282115e8

      Jan 27 08:58:25 RHEL-LEX kernel: R10: 0000033b07c0417c R11: 0000000000000001 R12: 0000000100319140

      Jan 27 08:58:25 RHEL-LEX kernel: R13: 0000000000000000 R14: 000000018109f793 R15: ffffffff81a01df8

      Jan 27 08:58:25 RHEL-LEX kernel: FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000

      Jan 27 08:58:25 RHEL-LEX kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b

      Jan 27 08:58:25 RHEL-LEX kernel: CR2: 0000003ce418e4f8 CR3: 000000010c79a000 CR4: 00000000000006f0

      Jan 27 08:58:25 RHEL-LEX kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

      Jan 27 08:58:25 RHEL-LEX kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

      Jan 27 08:58:25 RHEL-LEX kernel: Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a8d020)

      Jan 27 08:58:25 RHEL-LEX kernel: Stack:

      Jan 27 08:58:25 RHEL-LEX kernel: ffff880037aecaa0 0000033a2ffb8800 0000033a2dd63700 0000000100319140

      Jan 27 08:58:25 RHEL-LEX kernel: <d> 00000000b7eef3f8 000000010031911c ffffffff81a01ec8 ffffffff81a01fd8

      Jan 27 08:58:25 RHEL-LEX kernel: <d> ffffffff81c06680 0000000000000000 ffffffffffffffff ffffffff81de1000

      Jan 27 08:58:25 RHEL-LEX kernel: Call Trace:

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81009f89>] ? cpu_idle+0x79/0x110

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff8150cbea>] ? rest_init+0x7a/0x80

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c26f8f>] ? start_kernel+0x424/0x430

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124

      Jan 27 08:58:25 RHEL-LEX kernel: Code: fc ff 48 8b 45 b8 48 89 83 d8 00 00 00 48 8b 55 c8 48 89 93 d0 00 00 00 49 8b 46 78 4c 29 e8 48 89 83 c8 00 00 00 4c 89 ff 57 9d <0f> 1f 44 00 00 48 8b 5d d8 4c 8b 65 e0 4c 8b 6d e8 4c 8b 75 f0

      Jan 27 08:58:25 RHEL-LEX kernel: Call Trace:

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81009f89>] ? cpu_idle+0x79/0x110

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff8150cbea>] ? rest_init+0x7a/0x80

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c26f8f>] ? start_kernel+0x424/0x430

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129

      Jan 27 08:58:25 RHEL-LEX kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124

      Jan 27 08:58:25 RHEL-LEX kernel: ------------[ cut here ]------------

       

      Messages Seen at DOM0 .

       

      1. xend.log

       

       

      [2014-01-27 10:35:05 5910] WARNING (XendDomainInfo:2131) Domain has crashed: name=0004fb000006000080c3bb7a0c57cfc6 id=7.

      [2014-01-27 10:35:05 5910] DEBUG (XendDomainInfo:3145) XendDomainInfo.destroy: domid=7

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2473) Destroying device model

      [2014-01-27 10:35:07 5910] INFO (image:614) 0004fb000006000080c3bb7a0c57cfc6 device model terminated

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2480) Releasing devices

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vif/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vif/1

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/1

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vbd/51728

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vbd/51744

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vfb/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing console/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2478) No device model

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2480) Releasing devices

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:2486) Removing vif/0

      [2014-01-27 10:35:07 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2486) Removing vif/1

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/1

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2486) Removing vbd/51728

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2486) Removing vbd/51744

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:1287) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:128) XendDomainInfo.create_from_dict({'vcpus_params': {'cap': 0, 'weight': 27500}, 'PV_args': '', 'features': '', 'cpus': [[], [], [], []], 'paused': 0, 'use_tmp_kernel': False, 'actions_after_reboot': 'restart', 'memory_sharing': 0, 'superpages': 0, 'VCPUs_live': 1, 'PV_bootloader': '', 'actions_after_crash': 'restart', 'vbd_refs': ['17fa4b30-7a39-6000-108e-527bf16f55c6', '77fb5f0d-d869-5c8e-46a3-29ed78883a2c'], 'PV_ramdisk': '', 'memory_dynamic_min': 4294967296, 'name_label': '0004fb000006000080c3bb7a0c57cfc6', 'VCPUs_at_startup': 1, 'HVM_boot_params': {'order': 'cn'}, 'platform': {'videoram': 4, 'hpet': 0, 'stdvga': 0, 'vnclisten': '127.0.0.1', 'loader': '/usr/lib/xen/boot/hvmloader', 'rtc_timeoffset': 0, 'serial': 'pty', 'vncunused': 1, 'usb': 0, 'boot': 'cn', 'xen_platform_pci': 1, 'pci': [], 'pae': 1, 'vpt_align': 1, 'hap': 1, 'viridian': 0, 'acpi': 1, 'localtime': 0, 'timer_mode': 1, 'vnc': 1, 'nographic': 0, 'pci_msitranslate': 1, 'oos': 1, 'apic': 1, 'nomigrate': 0, 'xauthority': '//.Xauthority', 'tsc_mode': 0, 'guest_os_type': 'linux', 'device_model': '/usr/lib/xen/bin/qemu-dm', 'keymap': 'en-us', 'pci_power_mgmt': 0, 'isa': 0}, 'PV_kernel': '', 'console_refs': ['4d010487-d973-47ca-cfad-54f699ebb097', 'fd3997d5-17ca-98fb-e498-d6b997f47666'], 'online_vcpus': 1, 'acpi_s3': 0, 'blocked': 0, 'acpi_s4': 0, 'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shutdown': 0, 'HVM_boot_policy': 'BIOS order', 'description': '', 'shutdown_reason': 3, 'VCPUs_max': 4, 'start_time': 1390793309.466073, 'memory_static_max': 4294967296, 'expose_host_uuid': 0, 'use_tmp_ramdisk': False, 'actions_after_shutdown': 'destroy', 'crashed': 0, 'memory_dynamic_max': 4294967296, 'actions_after_suspend': '', 'is_a_template': False, 'PV_bootloader_args': '', 'is_control_domain': False, 'uuid': '0004fb00-0006-0000-80c3-bb7a0c57cfc6', 'cpu_time': 78.093066777999994, 'shadow_memory': 36, 'memory_static_min': 0, 'dying': 0, 'vcpu_avail': 15L, 'on_xend_start': 'ignore', 'other_config': {}, 'auto_power_on': False, 'running': 0, 'domid': 7, 'Description': '', 'vif_refs': ['f9ffef51-fb1d-2ed5-7221-142e219a7825', '74b204a2-473b-ca41-f428-0697fc5ca775'], 'target': 0, 'vtpm_refs': [], 's3_integrity': 1, 'devices': {'74b204a2-473b-ca41-f428-0697fc5ca775': ('vif', {'bridge': '103d603463', 'mac': '00:21:f6:00:00:05', 'devid': 1, 'uuid': '74b204a2-473b-ca41-f428-0697fc5ca775'}), '17fa4b30-7a39-6000-108e-527bf16f55c6': ('vbd', {'uuid': '17fa4b30-7a39-6000-108e-527bf16f55c6', 'bootable': 1, 'devid': 51728, 'driver': 'paravirtualised', 'dev': 'xvdb', 'uname': 'file:/OVS/Repositories/0004fb00000300009b0e5ca6781c1efa/VirtualDisks/0004fb0000120000cd7ba3b554b48912.img', 'mode': 'w'}), 'fd3997d5-17ca-98fb-e498-d6b997f47666': ('console', {'location': '6', 'devid': 0, 'protocol': 'vt100', 'uuid': 'fd3997d5-17ca-98fb-e498-d6b997f47666', 'other_config': {}}), 'f9ffef51-fb1d-2ed5-7221-142e219a7825': ('vif', {'bridge': '10840b7e14', 'mac': '00:21:f6:00:00:04', 'devid': 0, 'uuid': 'f9ffef51-fb1d-2ed5-7221-142e219a7825'}), '77fb5f0d-d869-5c8e-46a3-29ed78883a2c': ('vbd', {'uuid': '77fb5f0d-d869-5c8e-46a3-29ed78883a2c', 'bootable': 0, 'devid': 51744, 'driver': 'paravirtualised', 'dev': 'xvdc', 'uname': 'phy:/dev/mapper/360a9800065336563465a4c6b736c7654', 'mode': 'w'}), '4d010487-d973-47ca-cfad-54f699ebb097': ('vfb', {'vncunused': 1, 'other_config': {'vncunused': 1, 'vnclisten': '127.0.0.1', 'vnc': '1', 'keymap': 'en-us'}, 'vnc': '1', 'uuid': '4d010487-d973-47ca-cfad-54f699ebb097', 'vnclisten': '127.0.0.1', 'keymap': 'en-us', 'location': '127.0.0.1:5900', 'devid': 0})}, 'notes': {'SUSPEND_CANCEL': 1}})

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2570) XendDomainInfo.constructDomain

      [2014-01-27 10:35:08 5910] DEBUG (XendDomain:476) Adding Domain: 8

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2908) XendDomainInfo.initDomain: 8 27500

      [2014-01-27 10:35:08 5910] DEBUG (image:336) No VNC passwd configured for vfb access

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: boot, val: cn

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: fda, val: None

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: fdb, val: None

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: soundhw, val: None

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: localtime, val: 0

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: serial, val: ['pty']

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: std-vga, val: 0

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: isa, val: 0

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: acpi, val: 1

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: usb, val: 0

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: usbdevice, val: None

      [2014-01-27 10:35:08 5910] DEBUG (image:890) args: gfx_passthru, val: None

      [2014-01-27 10:35:08 5910] INFO (image:821) Need to create platform device.[domid:8]

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:2935) _initDomain:shadow_memory=0x24, memory_static_max=0x100000000, memory_static_min=0x0.

      [2014-01-27 10:35:08 5910] INFO (image:179) buildDomain os=hvm dom=8 vcpus=4

      [2014-01-27 10:35:08 5910] DEBUG (image:948) domid          = 8

      [2014-01-27 10:35:08 5910] DEBUG (image:949) image          = /usr/lib/xen/boot/hvmloader

      [2014-01-27 10:35:08 5910] DEBUG (image:950) store_evtchn   = 5

      [2014-01-27 10:35:08 5910] DEBUG (image:951) memsize        = 4096

      [2014-01-27 10:35:08 5910] DEBUG (image:952) target         = 4096

      [2014-01-27 10:35:08 5910] DEBUG (image:953) vcpus          = 4

      [2014-01-27 10:35:08 5910] DEBUG (image:954) vcpu_avail     = 15

      [2014-01-27 10:35:08 5910] DEBUG (image:955) acpi           = 1

      [2014-01-27 10:35:08 5910] DEBUG (image:956) apic           = 1

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: vfb : {'vncunused': 1, 'other_config': {'vncunused': 1, 'vnclisten': '127.0.0.1', 'vnc': '1', 'keymap': 'en-us'}, 'vnc': '1', 'uuid': '4d010487-d973-47ca-cfad-54f699ebb097', 'vnclisten': '127.0.0.1', 'keymap': 'en-us', 'location': '127.0.0.1:5900', 'devid': 0}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/8/0'} to /local/domain/8/device/vfb/0.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'vncunused': '1', 'domain': '0004fb000006000080c3bb7a0c57cfc6', 'frontend': '/local/domain/8/device/vfb/0', 'uuid': '4d010487-d973-47ca-cfad-54f699ebb097', 'frontend-id': '8', 'vnclisten': '127.0.0.1', 'state': '1', 'keymap': 'en-us', 'location': '127.0.0.1:5900', 'online': '1', 'vnc': '1'} to /local/domain/0/backend/vfb/8/0.

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: console : {'location': '6', 'devid': 0, 'protocol': 'vt100', 'uuid': 'fd3997d5-17ca-98fb-e498-d6b997f47666', 'other_config': {}}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/8/0'} to /local/domain/8/device/console/0.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'domain': '0004fb000006000080c3bb7a0c57cfc6', 'frontend': '/local/domain/8/device/console/0', 'uuid': 'fd3997d5-17ca-98fb-e498-d6b997f47666', 'frontend-id': '8', 'state': '1', 'location': '6', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/8/0.

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: vbd : {'uuid': '17fa4b30-7a39-6000-108e-527bf16f55c6', 'bootable': 1, 'devid': 51728, 'driver': 'paravirtualised', 'dev': 'xvdb', 'uname': 'file:/OVS/Repositories/0004fb00000300009b0e5ca6781c1efa/VirtualDisks/0004fb0000120000cd7ba3b554b48912.img', 'mode': 'w'}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '51728', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/8/51728'} to /local/domain/8/device/vbd/51728.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'domain': '0004fb000006000080c3bb7a0c57cfc6', 'frontend': '/local/domain/8/device/vbd/51728', 'uuid': '17fa4b30-7a39-6000-108e-527bf16f55c6', 'bootable': '1', 'dev': 'xvdb', 'state': '1', 'params': '/OVS/Repositories/0004fb00000300009b0e5ca6781c1efa/VirtualDisks/0004fb0000120000cd7ba3b554b48912.img', 'mode': 'w', 'online': '1', 'frontend-id': '8', 'type': 'file'} to /local/domain/0/backend/vbd/8/51728.

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: vbd : {'uuid': '77fb5f0d-d869-5c8e-46a3-29ed78883a2c', 'bootable': 0, 'devid': 51744, 'driver': 'paravirtualised', 'dev': 'xvdc', 'uname': 'phy:/dev/mapper/360a9800065336563465a4c6b736c7654', 'mode': 'w'}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '51744', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/8/51744'} to /local/domain/8/device/vbd/51744.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'domain': '0004fb000006000080c3bb7a0c57cfc6', 'frontend': '/local/domain/8/device/vbd/51744', 'uuid': '77fb5f0d-d869-5c8e-46a3-29ed78883a2c', 'bootable': '0', 'dev': 'xvdc', 'state': '1', 'params': '/dev/mapper/360a9800065336563465a4c6b736c7654', 'mode': 'w', 'online': '1', 'frontend-id': '8', 'type': 'phy'} to /local/domain/0/backend/vbd/8/51744.

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: vif : {'bridge': '10840b7e14', 'mac': '00:21:f6:00:00:04', 'devid': 0, 'uuid': 'f9ffef51-fb1d-2ed5-7221-142e219a7825'}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:21:f6:00:00:04', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/8/0'} to /local/domain/8/device/vif/0.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'bridge': '10840b7e14', 'domain': '0004fb000006000080c3bb7a0c57cfc6', 'handle': '0', 'uuid': 'f9ffef51-fb1d-2ed5-7221-142e219a7825', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:21:f6:00:00:04', 'frontend-id': '8', 'state': '1', 'online': '1', 'frontend': '/local/domain/8/device/vif/0'} to /local/domain/0/backend/vif/8/0.

      [2014-01-27 10:35:08 5910] INFO (XendDomainInfo:2427) createDevice: vif : {'bridge': '103d603463', 'mac': '00:21:f6:00:00:05', 'devid': 1, 'uuid': '74b204a2-473b-ca41-f428-0697fc5ca775'}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:21:f6:00:00:05', 'handle': '1', 'state': '1', 'backend': '/local/domain/0/backend/vif/8/1'} to /local/domain/8/device/vif/1.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:97) DevController: writing {'bridge': '103d603463', 'domain': '0004fb000006000080c3bb7a0c57cfc6', 'handle': '1', 'uuid': '74b204a2-473b-ca41-f428-0697fc5ca775', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:21:f6:00:00:05', 'frontend-id': '8', 'state': '1', 'online': '1', 'frontend': '/local/domain/8/device/vif/1'} to /local/domain/0/backend/vif/8/1.

      [2014-01-27 10:35:08 5910] INFO (image:415) spawning device models: /usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '8', '-domain-name', '0004fb000006000080c3bb7a0c57cfc6', '-videoram', '4', '-k', 'en-us', '-vnc', '127.0.0.1:0', '-vncunused', '-vcpus', '4', '-vcpu_avail', '0xFL', '-boot', 'cn', '-serial', 'pty', '-acpi', '-net', 'nic,vlan=1,macaddr=00:21:f6:00:00:04,model=rtl8139', '-net', 'tap,vlan=1,ifname=tap8.0,bridge=10840b7e14', '-net', 'nic,vlan=2,macaddr=00:21:f6:00:00:05,model=rtl8139', '-net', 'tap,vlan=2,ifname=tap8.1,bridge=103d603463', '-M', 'xenfv']

      [2014-01-27 10:35:08 5910] INFO (image:464) device model pid: 27532

      [2014-01-27 10:35:08 5910] INFO (image:589) waiting for sentinel_fifo

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:3516) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '36', 'uuid': '0004fb00-0006-0000-80c3-bb7a0c57cfc6', 'on_reboot': 'restart', 'start_time': '1390799108.45', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'image': "(hvm (kernel '') (expose_host_uuid 0) (superpages 0) (tsc_mode 0) (videoram 4) (hpet 0) (boot cn) (vnclisten 127.0.0.1) (loader /usr/lib/xen/boot/hvmloader) (serial pty) (vpt_align 1) (xen_platform_pci 1) (vncunused 1) (rtc_timeoffset 0) (pci ()) (pae 1) (stdvga 0) (hap 1) (viridian 0) (acpi 1) (localtime 0) (timer_mode 1) (vnc 1) (nographic 0) (guest_os_type linux) (pci_msitranslate 1) (oos 1) (apic 1) (nomigrate 0) (device_model /usr/lib/xen/bin/qemu-dm) (keymap en-us) (pci_power_mgmt 0) (usb 0) (xauthority //.Xauthority) (isa 0) (notes (SUSPEND_CANCEL 1)))", 'name': '0004fb000006000080c3bb7a0c57cfc6'}

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:1864) Storing domain details: {'console/port': '6', 'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'store/port': '5', 'cpu/2/availability': 'online', 'vm': '/vm/0004fb00-0006-0000-80c3-bb7a0c57cfc6', 'domid': '8', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'memory/target': '4194304', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '1044476', 'cpu/1/availability': 'online', 'console/type': 'ioemu', 'name': '0004fb000006000080c3bb7a0c57cfc6'}

      [2014-01-27 10:35:08 5910] DEBUG (DevController:139) Waiting for devices vif2.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:139) Waiting for devices vif.

      [2014-01-27 10:35:08 5910] DEBUG (DevController:144) Waiting for 0.

      [2014-01-27 10:35:08 5910] DEBUG (XendDomainInfo:1951) XendDomainInfo.handleShutdownWatch

      [2014-01-27 10:35:08 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/8/0/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/8/0/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:643) hotplugStatusCallback 1.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:144) Waiting for 1.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/8/1/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/8/1/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:643) hotplugStatusCallback 1.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vscsi.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vbd.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:144) Waiting for 51728.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/8/51728/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:643) hotplugStatusCallback 1.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:144) Waiting for 51744.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/8/51744/hotplug-status.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:643) hotplugStatusCallback 1.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices ioports.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices irq.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vkbd.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vfb.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices console.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:144) Waiting for 0.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices pci.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices tap2.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices tap.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vusb.

      [2014-01-27 10:35:09 5910] DEBUG (DevController:139) Waiting for devices vtpm.