Forum Stats

  • 3,770,108 Users
  • 2,253,066 Discussions
  • 7,875,315 Comments

Discussions

Oracle T8 -4 Server Networking best practice.

User_5M0EJ
User_5M0EJ Member Posts: 13 Green Ribbon

Experts,

Designing Oracle T8-4 networking , have 4 onboard NIC and 2 dual port NIC 10G card installed .

CDOM & IDOM configured.

Want to distribute the bandwidth for Data ,heartbeat, Management ports.

Once I create aggregation aggr0 for ( 2 Ports for one NIC ) .


1 . Is it possible to create multiple Virtual switch on aggr0 to segregate bandwidth for each workload

2 . In case only one Virtual switch possible Is it possible to limit network bandwidth on virtual NIC's for each workload.

3 . Is there any other best practice for cluster interconnect ,because the same traffic going via 10G interface based on tagged VLAN.

Regards,

Satya

Answers

  • Nik
    Nik Member Posts: 2,840 Bronze Crown

    Hi.

    Please clear:

    1. What Solaris version You use at Control an Guest Domains? Different Solaris version give different features.
    2. What type of Cluster You use? ( Oracle Solaris Cluster or Oracle Real Application Cluster).
    3. What general topology ? (You have many T8-4 server and build cluster between this servers or between guest domains on different server or some else).


    General recommendation - use separate Hi-Speed NIC for every type of workloads.

    But only You can know what performance really required.

    Regards,

    Nik

  • User_5M0EJ
    User_5M0EJ Member Posts: 13 Green Ribbon

    Hi Nik,


    1. What Solaris version You use at Control an Guest Domains? Different Solaris version give different features.

    Solaris 11

    1. What type of Cluster You use? ( Oracle Solaris Cluster or Oracle Real Application Cluster).

    Oracle Solaris Cluster

    1. What general topology ? (You have many T8-4 server and build cluster between this servers or between guest domains on different server or some else).

    Its between 2( T8-4 ) in Global zone level - Local zones are part of RG


    Regards,

    Satya

  • Nik
    Nik Member Posts: 2,840 Bronze Crown

    Hi.

    1 You can not share physical link between virtual switches. (IMHO). You can create many virtual switches that used different physical ports.

    2 You can limit bandwidth for link. https://docs.oracle.com/cd/E80106_01/html/E80109/controlnetworkbandwidth.html

    3 Solaris Cluster interconnect require at least 2 separate links. This links can share physical data-links, but use separate VLANs.


    Is not clear for me: How many Solaris Clusters You plan use ?

    T8-4 -> Split to LDOM (Cluster Node) -> Split to local Zones (Many Clusters)

      T8-4 (Cluster Node) -> Split to local Zones (Single Cluster)


    Regards,

    Nik

  • User_5M0EJ
    User_5M0EJ Member Posts: 13 Green Ribbon

    Thanks, much Nik for your time.



    1 ] You can not share the physical links between virtual switches. (IMHO). You can create many virtual switches that used different physical ports.

    As per the above draft, it's not possible to create multiple vswitch - Is my understanding was correct.

    In such a case, if I can able to create one vswitch - I can limit bandwidth as per  https://docs.oracle.com/cd/E80106_01/html/E80109/controlnetworkbandwidth.html



    3]

    Is not clear for me: How many Solaris Clusters You plan use ?

    T8-4 -> Split to LDOM (Cluster Node) -> Split to local Zones (Many Clusters)

      T8-4 (Cluster Node) -> Split to local Zones (Single Cluster)


    The setup is as bleow

    Cluster between Global Zones and Local Zones are part of Resource Group for Cluster.

    In this case, how can I plan my heartbeat link as virtual Interconnect (vnet )with VLAN ID tagged?



  • Nik
    Nik Member Posts: 2,840 Bronze Crown

    Hi.

    Not reason clustering Control and IO Domains.

    You should clustering only domains that run some Data-services.


    Reason for create IO-domains- for increasing availability of guest domains.

    So IO-domain should provide independent path for disk and network resources.

    As You plan use only one network aggregation - Are You sure that You need create separate IO Domain ?


    For every cluster You should use separate VLAN for every cluster interconnect link.


    Regards,

    Nik

  • User_5M0EJ
    User_5M0EJ Member Posts: 13 Green Ribbon

    Thank you Nik

    How I can create redundancy between Control Domain and IO domain in the above setup while configuring the IDOM . I mean distribution of CPU Memory and PCI , FC

    T8-4 ( 1 CDOM = 1 IDOM)


    Regards

    Satya

  • Nik
    Nik Member Posts: 2,840 Bronze Crown
    edited Nov 8, 2021 3:43PM

    Hi.

    Recommended best practice: https://www.oracle.com/technetwork/server-storage/vm/ovmsparc-best-practices-2334546.pdf



    https://www.oracle.com/a/ocom/docs/sparc-t8-m8-server-architecture.pdf page 22, 23

    Control and IO Domain should use different Root-Complex ( Different IO controllers - prefer).


    Share current configuration of T8-4 server.

    (Type and number of network cards, type and number of FC-cards , RAM, number of CPU, number of internal disks).


    Regards,

    Nik