This discussion is archived
1 2 Previous Next 22 Replies Latest reply: Mar 15, 2012 3:12 AM by Avi Miller RSS

hard partitioning and hyperthreading and licensing and confusion

Martin Brambley Newbie
Currently Being Moderated
this oracle paper http://www.oracle.com/technetwork/topics/virtualization/ovm-hardpart-167739.pdf says that for hard partitioning we have to edit the vm.cfg file and other messages i have posted here tell me that this still enables HA but only disables live migration and the like

but if i have hyperthreading enabled - which i have , a 40 core server DL580 where vmserver and vmmanager show 80 cores

if i edit vm.cfg to to enable hard partitioning and say cpus = '0-3' does that fix the guest to using 4 cores (which would be 8 out of the 80 recognised by ovm) or 4 of the entities seen as cores to ovm (4 threads out of the 80)


licensing...

the main reason the business has chosen ovm over vmware is the ability to hard partition saving us a fortune on oracle licensing (well support - weve already bought the licenses) i have databases running on 24 cores where the server is >95 idle at peak! there is much documentation around (http://www.oracle.com/technetwork/database/clustering/oracle-rac-in-oracle-vm-environment-131948.pdf) and the now removed (partitioning.pdf) stating that oracle will allow you to license a subset of the cores via hard partitioning (unlike vmware) - although as pointed out in OVM 3.01 and hard partitioning
the new doc states that editing vm.cfg is not supported

so whats the deal? do i have to revert to ovm2?

can you confirm?



shrinking power.....

if i have a 10 core server without HT turned on
and i have 2 guests configured to use 5 processors
if i then enable HT so that ovm then reports there are 20 cores available
has the effective CPU capacity of my guests just been reduced as they still say 5 processors (out of the now 20 reported for the vmserver?)

thanks
  • 1. Re: hard partitioning and hyperthreading and licensing and confusion
    Martin Brambley Newbie
    Currently Being Moderated
    on metalink Oracle VM General Policy Description [ID 466538.1] states :

    Hard partitioning with Oracle VM Server for x86

    With Oracle VM server for x86, guests are allowed to be pinned or bound to specific physical cores in a single Oracle VM server, see "Hard Partitioning with Oracle VM <http://www.oracle.com/technetwork/topics/virtualization/ovm-hardpart-167739.pdf>". Once hard partitioned, the VM guest application only needs to be licensed for the number of physical cores it is pinned to, see Oracle's licensing policies for partitioned environments <http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf> . Note that once a VM guest is pinned or bound to physical cores within a server, Oracle licensing restricts that guest from being live migrated to another VM server in the pool.
    so i guess the doc stating that editing the vm.cfg is not supported is a doc bug - can this be fixed?
  • 2. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    Martin Brambley wrote:
    if i edit vm.cfg to to enable hard partitioning and say cpus = '0-3' does that fix the guest to using 4 cores (which would be 8 out of the 80 recognised by ovm) or 4 of the entities seen as cores to ovm (4 threads out of the 80)
    If you edit vm.cfg on an HT-enabled machine, with cpus = '0-3', you'll be running on 2 cores, 4 threads. So, you can set it to 0-7 to run on 4 cores/8 threads. This is a bonus with hyperthreading: because we license by core, not thread, you can effectively double the vCPU count of your guests with the same core licensing. Another reason not to disable hyperthreading. :)
    the new doc states that editing vm.cfg is not supported
    The documentation will be updated with the release of 3.0.3 to specifically allow editing of vm.cfg for hard partitioning purposes.
    so whats the deal? do i have to revert to ovm2?
    No.
    if i have a 10 core server without HT turned on
    and i have 2 guests configured to use 5 processors
    if i then enable HT so that ovm then reports there are 20 cores available
    has the effective CPU capacity of my guests just been reduced as they still say 5 processors (out of the now 20 reported for the vmserver?)
    The effective CPU capacity of your guests has not been reduced, but your overall capacity has increased. If you disable HT, then the server was only running a single thread per core anyway. So, if you enable HT without changing your guests, you're still only running on a single thread per core. You can now double your vCPU counts in the guests to 10 and still sit within your license requirements (as you're still only using 10 cores on the physical server).
  • 3. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    Martin Brambley wrote:
    so i guess the doc stating that editing the vm.cfg is not supported is a doc bug - can this be fixed?
    It already has been fixed. The fix will be published with 3.0.3.

    Keep in mind that hard partitioning requires that you pin the CPUs for guests running the same products to the same cores, because you're licensing a subset of the cores of the machine.

    So, let's take your example of a server with 40 cores and 80 threads, showing 80 logical CPUs. If you only license 10 cores worth of database, you have to pin all DB guests to the same 10 cores (i.e. 20 logical vCPUs with HT-enabled) on that machine. The other 30 cores/60 threads can be used to do other things, but they cannot run database. If this machine is only going to run DB VMs, you will effectively be wasting 30 cores.

    Just wanted to make sure that was explicitly clear.
  • 4. Re: hard partitioning and hyperthreading and licensing and confusion
    821215 Newbie
    Currently Being Moderated
    I recall someone mentioning some time ago that the way the virtual cores are numbered can, depending on CPU model, not necessarily be obvious?

    With HT on, will vcpus 0-1 be the two threads on core 0? Or could, say, vcpus 0+8 be the two threads of core 0?
  • 5. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    user10786594 wrote:
    With HT on, will vcpus 0-1 be the two threads on core 0? Or could, say, vcpus 0+8 be the two threads of core 0?
    I believe with OVM3, the threads/cores are numbered sequentially, so 0-1 are threads on core 0 and 2-3 are threads on core 1. However, I'm not 100% sure, so I'm double-checking with the development team.

    The other thing you want to do is ensure you pin to cores within the same NUMA node. You can use xm info to display information about node_to_cpu and node_to_memory mappings. Use this information to ensure that the vCPUs you use for pinning do not span NUMA nodes. This ensures the best performance.
  • 6. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    Seems I was a bit wrong on sequential numbering, but at least xm info tells us what we need to know:
    user10786594 wrote:
    I recall someone mentioning some time ago that the way the virtual cores are numbered can, depending on CPU model, not necessarily be obvious?
    xm info on OVM3 shows you the node (socket) and core/thread mapping. Here's an Intel Westmere-EP box with 2 sockets:
     nr_cpus                : 24
    nr_nodes               : 2
    cores_per_socket       : 6
    threads_per_core       : 2
    ...
    node_to_cpu            : node0:0-5,12-17
                             node1:6-11,18-23
    So, cpus 0-5 are the first threads on the first 6 cores of socket 0. The second 6 threads on socket 1 are numbered 12-17. On node 1, we have 6-11 being the first six threads and 18-23 being the second.

    Here's an example from a 4-socket AMD box, which shows non-sequential numbering:
    node_to_cpu            : node0:0,4,8,12,16,20
                             node1:1,5,9,13,17,21
                             node2:2,6,10,14,18,22
                             node3:3,7,11,15,19,23
    The development guys tell me this numbering is stable for a given machine, but may not be stable between machines.
  • 7. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    I've just been introduced to the most amazing tool:
     # xenpm get-cpu-topology 
    This will show you the exact logical CPU to core/socket mapping:
    CPU     core     socket
    CPU0      0      0
    CPU1      1      0
    CPU2      2      0
    CPU3      8      0
    CPU4      9      0
    CPU5      10      0
    CPU6      0      1
    CPU7      1      1
    CPU8      2      1
    CPU9      8      1
    CPU10      9      1
    CPU11      10      1
    CPU12      0      0
    CPU13      1      0
    CPU14      2      0
    CPU15      8      0
    CPU16      9      0
    CPU17      10      0
    CPU18      0      1
    CPU19      1      1
    CPU20      2      1
    CPU21      8      1
    CPU22      9      1
    CPU23      10      1
  • 8. Re: hard partitioning and hyperthreading and licensing and confusion
    821215 Newbie
    Currently Being Moderated
    Excellent - thanks Avi.
  • 9. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    I've also been told that because the guests are not in fact aware of the threads/cores, it's better to pin to different cores, not multiple threads on the same core. But, licensing only allows you to pin to a specific number of cores. What you could do is pin one guest to Thread0 of 8 cores and another guest to Thread1 of the same 8 cores. This only requires 8 cores of licensing, but means that the guests can run on different threads at the same time.

    Obviously, any pinning has some performance impact. You're better off doing your own experimentation on your own hardware to determine the most efficient combination of pinning to cores to determine the best/most efficient pinning strategy for your workload(s).
  • 10. Re: hard partitioning and hyperthreading and licensing and confusion
    Martin Brambley Newbie
    Currently Being Moderated
    Thanks Avi - answered in full

    Cheers
  • 11. Re: hard partitioning and hyperthreading and licensing and confusion
    user386688 Newbie
    Currently Being Moderated
    Hi and sorry to hijack the thread, but...

    On the subject of hard-partitioning licensing a subset of cores of a physical OVM server, and pinning all VMs using the product to those same cores, where you are running a pool with multiple VM servers does this not suggest that you should also be using a placement policy to run them on the same physical server? Otherwise you would be licensing these same cores over all machines in the pool - 4 cores x 5 potential servers = 20 cores *0.5 processor factor = 10 cores.

    As far as I can see, placement policy has disappeared from OVM 3 and affinity and anti-affinity policies would not seem to offer the required functionality.

    Cheers (and Hi Martin if you read this)

    Julian
  • 12. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    Julian Mccoy-Daly wrote:
    On the subject of hard-partitioning licensing a subset of cores of a physical OVM server, and pinning all VMs using the product to those same cores, where you are running a pool with multiple VM servers does this not suggest that you should also be using a placement policy to run them on the same physical server? Otherwise you would be licensing these same cores over all machines in the pool - 4 cores x 5 potential servers = 20 cores *0.5 processor factor = 10 cores.
    Correct.
    As far as I can see, placement policy has disappeared from OVM 3 and affinity and anti-affinity policies would not seem to offer the required functionality.
    You can use storage repositories as a method of placement policy in OVM3: create a storage repository for those specific guests and only present it to a subset of servers in the pool. Then the guests in that repository can only start on those servers. And it doesn't have to be a large repository: just needs to store the vm.cfg files for the guests, not the virtual disk images.
  • 13. Re: hard partitioning and hyperthreading and licensing and confusion
    user386688 Newbie
    Currently Being Moderated
    Hi Avi,

    What you say makes sense, though http://docs.oracle.com/cd/E26996_01/e18549/vm_hardpart.html states:

    "Hard partitioning is the ONLY case where you are allowed to modify the virtual machine configuration file manually. For all other changes to a virtual machine, use the Oracle VM Manager user interface."

    And the implication here is that it is modifying the vm.cfg to add the pin statement, not to add repositories that are manually created with the VirtualMachines/VMUUID/vm.cfg that is then modified to point back to the vm image in a different repository. I suppose you probably would create a different repository containing the entirety of these VMs to avoid "breaking" this "rule". The whole UUID thing seems a bit of a kludge really. Xen is happy as long as it knows where to find the images, its just the manager can't operate without a vm name that is a uuid - makes operation from the xen command line a royal pain for such activities as xm dump-core, etc... and generally abhorrent for prehistoric command line users such as me :(

    What you say about hard partitioning makes a certain amount of sense to me, but:
    1) None of the Oracle docs I have seen suggest that you need to actually tie a vm to a particular server
    2) None of the other hard partitioning technologies would allow you to commit potentially many VMs to the same subset of CPUs - would they?

    It would seem a vehicle that would allow you to drive down your license costs by confining VMs to a licensed set of CPUs on 1 machine, which would generally present potential performance issues admittedly at a reduced license cost, while nobbling the HA/live migration capabilities of OVM. Much nicer would be if Oracle would elevate the potential for licensing of their own OVM such that pinning CPUs in the vm.cfg is the only requirement for ensuring you don't break your licenses, while allowing HA/live migration and the other new policies at OVM3 to make one's life as an admin generally easier rather than harder.

    In an increasingly virtual world, Oracle are giving with one hand while busily taking away with the other... C'mon - give us a reason to want to use OVM!
  • 14. Re: hard partitioning and hyperthreading and licensing and confusion
    Avi Miller Guru
    Currently Being Moderated
    Julian Mccoy-Daly wrote:
    "Hard partitioning is the ONLY case where you are allowed to modify the virtual machine configuration file manually. For all other changes to a virtual machine, use the Oracle VM Manager user interface."
    Correct.
    And the implication here is that it is modifying the vm.cfg to add the pin statement, not to add repositories that are manually created with the VirtualMachines/VMUUID/vm.cfg that is then modified to point back to the vm image in a different repository.
    Why would you need to do this? You can select which repository is used to store the configuration file on the first step of the Create VM wizard. Just select the repository that's only presented to the subset of servers. No manual modifications required at all.
    What you say about hard partitioning makes a certain amount of sense to me, but:
    1) None of the Oracle docs I have seen suggest that you need to actually tie a vm to a particular server
    They're tied to a physical CPU, which implies a physical server. As long as you're only using the CPUs you've paid for, you're OK. The onus is on you to ensure that you only use licensed CPUs, though.
    2) None of the other hard partitioning technologies would allow you to commit potentially many VMs to the same subset of CPUs - would they?
    Some of them do, but perhaps none as flexibly as Oracle VM for x86.
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points