11 Replies Latest reply: Sep 9, 2010 2:05 PM by 807557 RSS

    B6000 Mutli-Fabric NEM  question

    807557
      Hi all.
      I have B6000 cage with 2 X6270 (Sol10 x86 U8) and 2 VMF NEM (SP ver 2.0.3.12)
      My original intention was to use NEM0 for passthru and NEM1 for internal blade-to-blade connectivity but I really struggle with 10 GB virtual card.
      I do see hxge card on blades, but only hxge0, while I have 2 identical NEMs and those cards show disconnected status on both blades.
      I understand that NEM1 should appear as hxge1.
      The SFP is not connected to external device on none of them.
      Both blades see only hxge0 with status disconnected while igb cards are working OK with external switch.
      I do realize that this NEM is not a real switch blade but on the other hand it is stated clearly that it has
      In-chassis blade-to-blade communication and my local SUN representative insists it has switching capabilities.
      Can someone enlighten this issue or show me where did I go wrong?
      Thanks.
        • 1. Re: B6000 Mutli-Fabric NEM  question
          807557
          The VNEM is NOT a switch. It is individual ASICs that share a PHY connection (or two, rather, with failover between the two). You should be able to get communication between hosts, provided you do not need to route through an external gatway.

          However, one thing that confused me when first testing the VNEM was that you need to completely power cycle the SB6000 chassis before it becomes "ready." Did you do this?

          Charles
          • 2. Re: B6000 Mutli-Fabric NEM  question
            807557
            Thanks, Charles.
            In fact the VNEMs were inserted into the cage before assembling into rack. So that the whole cage was powered with the NEMs inside, if it matters.
            However I am even more confused than before.
            If it is not switch and doesn't do bridging, spanning tree and other stuff, how do my blades communicate between them and with what wire speed?
               1   igb0       147.234.157.135/20  mtu=1500 Up   conn. zone: global
               2   hxge0      10.255.255.10/24    mtu=1500 Up   disc. zone: global
            Mine always show "Disconnected".

            Does it mean that I need to purchase external switching device to transfer data between those blades inside the same cage?
            I connect to external networks with igb cards.
            • 3. Re: B6000 Mutli-Fabric NEM  question
              807557
              Correct you cannot communicate blade to blade internal to the VNEM you must go to an external switch to be able to communicate from blade to blade. The VNEM is not a switch and does not have this capability.
              • 4. Re: B6000 Mutli-Fabric NEM  question
                807557
                Great. Can you please comment following statement from Oracle site?

                [http://www.oracle.com/us/products/servers-storage/servers/blades/031039.htm]


                AT A GLANCE
                New Network Express Module for Sun Blade 6000 Modular Systems
                Lower cost 10GB performance
                Reduces cabling by 10x
                *{color:#ff0000}In-chassis blade-to-blade communication{color}*
                Zero management with network transparency
                Integrated SAS storage networking
                Supports Solaris OS, Linux, and Windows



                Edited by: Rony_Shykh on Jun 1, 2010 10:33 PM

                Edited by: Rony_Shykh on Jun 1, 2010 10:34 PM
                • 5. Re: B6000 Mutli-Fabric NEM  question
                  807557
                  Marc Hamilton posted a diagram once: http://blogs.sun.com/marchamilton/entry/more_on_sun_s_virtual

                  Maybe he can provide more insight, or at least point to the technical literature.

                  Charles
                  • 6. Re: B6000 Mutli-Fabric NEM  question
                    807557
                    I believe this is referring to the SAS expander communication between blade and disk blade not the 10GB ports. The SAS expander communication is blade to blade within the chassis but I would agree this is slightly misleading.
                    • 7. Re: B6000 Mutli-Fabric NEM  question
                      807557
                      Thank you guys.
                      It was really enlightening and helpful discussion.
                      Perhaps now I will be able to fulfil my need without spending on worthless hardware with misleading description.
                      I need to build a datacenter in a box with fastest internal communication between blades inside the cage.
                      And provide regular output of those blades for outside access.
                      That's why I originally bought 2 VNEMS. I wanted to use those hxge cards on one NEM for internal communication and igb of the other one for external.
                      What are my Sun options for this:
                      I would really like to avoid wasting another 10k.
                      Thanks again.
                      • 8. Re: B6000 Mutli-Fabric NEM  question
                        807557
                        Rony,

                        The Blade-to-Blade communication is supported in this NEM as long as the uplink is active.
                        Can you please email me directly to get further details on this.

                        thx
                        paula
                        • 9. Re: B6000 Mutli-Fabric NEM  question
                          807557
                          I double checked with our engineering team and they also confirm that blade-to-blade communication is supported within the chassis as long as the uplink is active as Paula stated. I apologize for the mis-information. My error in understanding as the documentation is not clear on the subject.
                          • 10. Re: B6000 Mutli-Fabric NEM  question
                            807557
                            Hi Paula,

                            Do you have a procedure to enable the blade-to-blade communication please? I have sifted through copious amounts of docs and systemhandbook, without any luck. Added with conflicting reports that it can or cannot be done. I am sitting with two blade6000 chassis' and 8 x6270 modules in each.

                            Any help would be hugely appreciated

                            Monz
                            • 11. Re: B6000 Mutli-Fabric NEM  question
                              807557
                              Monz:

                              My understanding of how this works is you must have external connection to a switch so the uplink is active for the blade-to-blade internal communication to work.

                              Regards,

                              Mick