7 Replies Latest reply: Feb 2, 2011 2:01 PM by Simon Haslam RSS

    LDOMs 2.0 & max-core/whole-core functionality

    bhudson
      It appears that on Solaris 11 Express (SPARC on T5240) the system/ldoms/ldomsmanager package does not work properly for the core assignment functionality.

      [root@ilhsf001h002]# ldm ls
      NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
      primary active -n-cv- SP 8 8G 17% 23h 27m
      ilhsf001v001 active -n---- 5000 8 8G 0.2% 21h 15m
      [root@ilhsf001h002]# ldm ls-constraints -p ilhsf001v001 | grep CONSTRAINT
      CONSTRAINT|cpu=whole-core|max-cores=1
      [root@ilhsf001h002]# ldm add-vcpu -c 1 ilhsf001v001
      The number of cores for domain ilhsf001v001 cannot exceed max-cores=1
      Resource addition failed
      [root@ilhsf001h002]# ldm add-vcpu 8 ilhsf001v001
      Cannot remove whole-core partitioning constraint from a bound or active domain
      [root@ilhsf001h002]# ldm set-vcpu -c 2 ilhsf001v001
      The number of cores for domain ilhsf001v001 cannot exceed max-cores=1
      Resource modification failed
      [root@ilhsf001h002]# ldm set-vcpu 16 ilhsf001v001
      Cannot remove whole-core partitioning constraint from a bound or active domain
      [root@ilhsf001h002]#
        • 1. Re: LDOMs 2.0 & max-core/whole-core functionality
          Honglin Su-Oracle
          It's the expected behavior in 2.0 since the max-cores was set to 1. It's not related to Solaris 10 or Solaris 11 Express.

          CPU Dynamic Reconfiguration

          The whole-core constraint is fully compatible with CPU dynamic reconfiguration (DR). When a domain is defined with the whole-core constraint, you can use the ldm add-vcpu -c, ldm set-vcpu -c, or remove-vcpu -c command to change the number of cores on an active domain.

          However, if a bound or active domain is not in delayed reconfiguration mode, its number of cores cannot exceed the maximum number of cores. This maximum is set with the maximum core constraint, which is automatically enabled when the whole-core constraint is enabled. Any CPU DR operation that does not satisfy the maximum core constraint fails.

          Please refer to 2.0 Admin Guide in details how you can change these settings.
          http://download.oracle.com/docs/cd/E19608-01/821-1485/cpuallocation/index.html
          • 2. Re: LDOMs 2.0 & max-core/whole-core functionality
            bhudson
            The inability to modify the max-core or whole-core constraint is essentially what I was referring to that does not work properly. If these values cannot be adjusted to a running (active) domain then by definition the feature has broken CPU DR. It means that if one assigns a whole-core to a domain, CPUs cannot then be added or removed from the running domain effectively breaking the 'DR' for vCPUs (due to the max-core constraint). I consider this a major bug and should be addressed as a CR. After all, what's the point of providing the feature if it still requires the guest OS to be taken down for the change. This is more akin to adding a new chip on a physical system.
            • 3. Re: LDOMs 2.0 & max-core/whole-core functionality
              Honglin Su-Oracle
              I'm aware of this and thank you for your input! You can contact me offline (first.last@oracle.com) if you want to discuss further.
              • 4. Re: LDOMs 2.0 & max-core/whole-core functionality
                Eric S
                I responded to bhudson separately; I'm copying that reply here for the benefit of the forum:

                The capabilities and limitations of the whole-core and max-core constraints are designed around having Oracle VM Server for SPARC technology meet Oracle's formal definition of hard-partitioning. Among other things, this precluded us from making the feature fully dynamic.

                So, to enable or disable the whole core constraint requires stopping and unbinding the domain first. And to change the max-cores value requires disabling the whole core constraint and then re-enabling it by specifying a higher core count, as the initial core count specified implicitly becomes the cap. In essence you should initially specify the highest core count you'll expect to use when you first enable the whole-core constraint. Then you'll be able to remove and add cores dynamically as long as you stay under that initial cap value.

                I hope this clarifies the issue.
                • 5. Re: LDOMs 2.0 & max-core/whole-core functionality
                  Simon Haslam
                  Eric

                  It is very interesting that you are suggesting that LDoms can be used as a means of hard-partitioning a server from an Oracle licence perspective.

                  Is this written down in any public Oracle documents yet? I can't see this specifically mentioned in [http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf], though possibly you could infer it (though "possibly" is not very appealing when it comes to licence audits!).

                  If LDoms, in certain configurations (i.e. with CPU Whole Core Allocation), can now be used to cap the Oracle processor licences required (and afterall Oracle will no doubt be keen to make it as attractive to run on T3 as on x86-64) then I think this needs to be explicitly stated, perhaps in the OVM licensing document ([http://www.oracle.com/technetwork/topics/virtualization/ovm-hardpart-167739.pdf]) as soon as possible.

                  Do you know if this is in progress as it sounds like it would be a straight-forward amendment?

                  TIA!


                  Simon
                  • 6. Re: LDOMs 2.0 & max-core/whole-core functionality
                    Eric S
                    Simon, I did not say that LDoms can be used as a hard partitioning technology. Such a licensing decision and change of designation has not received approval. What I did say is that we've done the engineering work to meet the requirements. It's now in the hands of the business/legal side of the house to decide on any future licensing changes for LDoms. I have little visibility into that process.

                    -Eric
                    • 7. Re: LDOMs 2.0 & max-core/whole-core functionality
                      Simon Haslam
                      Ah, OK. Thanks Eric - that was me with my enthusiastic techie hat on, jumping the gun! Let's hope the Oracle licensing/legal team do something about that then PDQ since the new T3s can be pretty big boxes.

                      Simon