Oracle T8 -4 Server Networking best practice.

User_5M0EJ

Experts,

Designing Oracle T8-4 networking , have 4 onboard NIC and 2 dual port NIC 10G card installed .

CDOM & IDOM configured.

Want to distribute the bandwidth for Data ,heartbeat, Management ports.

Once I create aggregation aggr0 for ( 2 Ports for one NIC ) .

1 . Is it possible to create multiple Virtual switch on aggr0 to segregate bandwidth for each workload

2 . In case only one Virtual switch possible Is it possible to limit network bandwidth on virtual NIC's for each workload.

3 . Is there any other best practice for cluster interconnect ,because the same traffic going via 10G interface based on tagged VLAN.

Regards,

Satya

Nik

Hi.

Please clear:

1. What Solaris version You use at Control an Guest Domains? Different Solaris version give different features.
2. What type of Cluster You use? ( Oracle Solaris Cluster or Oracle Real Application Cluster).
3. What general topology ? (You have many T8-4 server and build cluster between this servers or between guest domains on different server or some else).

General recommendation - use separate Hi-Speed NIC for every type of workloads.

But only You can know what performance really required.

Regards,

Nik

User_5M0EJ

Hi Nik,

1. What Solaris version You use at Control an Guest Domains? Different Solaris version give different features.

Solaris 11

1. What type of Cluster You use? ( Oracle Solaris Cluster or Oracle Real Application Cluster).

Oracle Solaris Cluster

1. What general topology ? (You have many T8-4 server and build cluster between this servers or between guest domains on different server or some else).

Its between 2( T8-4 ) in Global zone level - Local zones are part of RG

Regards,

Satya

Nik

Hi.

1 You can not share physical link between virtual switches. (IMHO). You can create many virtual switches that used different physical ports.

2 You can limit bandwidth for link. https://docs.oracle.com/cd/E80106_01/html/E80109/controlnetworkbandwidth.html

3 Solaris Cluster interconnect require at least 2 separate links. This links can share physical data-links, but use separate VLANs.

Is not clear for me: How many Solaris Clusters You plan use ?

T8-4 -> Split to LDOM (Cluster Node) -> Split to local Zones (Many Clusters)

  T8-4 (Cluster Node) -> Split to local Zones (Single Cluster)

https://docs.oracle.com/cd/E69294_01/html/E69310/bacfbadd.html#scrolltoc

Regards,

Nik

User_5M0EJ

Thanks, much Nik for your time.

1 ] You can not share the physical links between virtual switches. (IMHO). You can create many virtual switches that used different physical ports.

As per the above draft, it's not possible to create multiple vswitch - Is my understanding was correct.

In such a case, if I can able to create one vswitch - I can limit bandwidth as per  https://docs.oracle.com/cd/E80106_01/html/E80109/controlnetworkbandwidth.html

3]

Is not clear for me: How many Solaris Clusters You plan use ?

T8-4 -> Split to LDOM (Cluster Node) -> Split to local Zones (Many Clusters)

  T8-4 (Cluster Node) -> Split to local Zones (Single Cluster)

The setup is as bleow

Cluster between Global Zones and Local Zones are part of Resource Group for Cluster.

In this case, how can I plan my heartbeat link as virtual Interconnect (vnet )with VLAN ID tagged?

Nik

Hi.

Not reason clustering Control and IO Domains.

You should clustering only domains that run some Data-services.

Reason for create IO-domains- for increasing availability of guest domains.

So IO-domain should provide independent path for disk and network resources.

As You plan use only one network aggregation - Are You sure that You need create separate IO Domain ?

For every cluster You should use separate VLAN for every cluster interconnect link.

Regards,

Nik

User_5M0EJ

Thank you Nik

How I can create redundancy between Control Domain and IO domain in the above setup while configuring the IDOM . I mean distribution of CPU Memory and PCI , FC

T8-4 ( 1 CDOM = 1 IDOM)

Regards

Satya

Nik

Hi.

Recommended best practice: https://www.oracle.com/technetwork/server-storage/vm/ovmsparc-best-practices-2334546.pdf

https://www.oracle.com/a/ocom/docs/sparc-t8-m8-server-architecture.pdf page 22, 23

Control and IO Domain should use different Root-Complex ( Different IO controllers - prefer).

Share current configuration of T8-4 server.

(Type and number of network cards, type and number of FC-cards , RAM, number of CPU, number of internal disks).

Regards,

Nik