9 Replies Latest reply: Mar 21, 2013 11:04 AM by 997789 RSS

    [M3 x4170] Missing network interfaces after installation 2nd CPU

    997789
      Hi,

      I have an M3 x4170 Oracle machine with initially one CPU installed. Solaris has been installed and detects two (out of the four) network interfaces. This is correct as the other two interfaces are unusable due to the missing CPU.

      We have now moved this installation to a x4170 system with TWO CPUs installed. The additional CPU shows up, but the network interfaces do not.

      # prtconf -D | grep ixgb
      pci108e,484d, instance #0 (driver name: ixgbe)
      pci108e,484d, instance #1 (driver name: ixgbe)
      pci108e,484d (driver name: ixgbe)
      pci108e,484d (driver name: ixgbe)

      I have tried various combinations of 'touch /reconfigure; init6' and 'reboot -- -r' to have the system detect the new interfaces. This does not work.

      # grep ixgbe /etc/path_to_inst
      "/pci@0,0/pci8086,3c04@2/pci108e,484d@0" 0 "ixgbe"
      "/pci@0,0/pci8086,3c04@2/pci108e,484d@0,1" 1 "ixgbe"

      The pci devices seem to not get detected somehow. The path_to_inst file is missing references to the second controller. (Which is connected to the second CPU).

      Any ideas on how to resolve this issue? The whole reason for getting a two-cpu system is the additional 10Gbit interfaces..

      Thanks!
        • 1. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
          Reidod
          Hi,

          Can you post the output of 'dladm show-dev' command?

          Kind regards,
          Rei
          • 2. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
            997789
            Hi Reidod, thanks for your help!

            The dladm shows two interfaces only, unfortunately:
            dladm show-dev
            ixgbe0 link: up speed: 1000 Mbps duplex: full
            ixgbe1 link: up speed: 1000 Mbps duplex: full
            My impression is that during Solaris installation the system detected that only one CPU was installed, and has written a config file/setting somewhere to disable the third and fourth interface. Now that we have added the second CPU that file is not updated. Its the only conclusion I can come up with since the device reconfigure steps do not change the situation. But where this file could be...

            Edited by: 994786 on Mar 20, 2013 6:39 AM

            Edited by: 994786 on Mar 20, 2013 6:39 AM
            • 3. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
              Reidod
              Hi,

              I would suggest to do a reconfiguration boot (append -r at kernel command in GRUB) or give 'devfsadm -v'. After that try to plumb the interfaces:

              ifconfig ixgbe2 plumb
              ifconfig ixgbe3 plumb

              Check if the interfaces will be plumbed. The driver for the NIC is located under /kernel/drv directory.

              Regards,
              Rei
              • 4. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                997789
                Hi Reidod,

                The devfsadm -v, reboot -- -r or touch /reconfigure should all kind of do the same thing, redetect the device tree. However when I do the /etc/path_to_inst is even updated. And therefore the additional devices are not loaded.

                A driver issue would not be expected, as the first two interfaces use the exact same driver as the second two. Therefore the driver is loaded and working.

                Since the low level devices are not available in /devices nor /dev I cannot plumb the interfaces yet, unfortunately..
                • 5. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                  800381
                  What does a verbose boot (reboot -- -r -v) show?

                  What's the output from /usr/X11/bin/scanpci?

                  Do all four network interfaces show on either of those?
                  • 6. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                    Reidod
                    Can you try to unconfigure/configure the PCI card using cfgadm?


                    Regards,
                    Reinaldo
                    • 7. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                      997789
                      Thanks for your replies :)

                      user5287726, scanpci is not available on this server - I do not use X11 at all I believe. The next best thing, prtdiag, shows all controllers:

                      ==== On-Board Devices ======
                      Onboard Video
                      X540 10GbE Controller
                      X540 10GbE Controller
                      X540 10GbE Controller
                      X540 10GbE Controller
                      Intel C600 Series - SAS
                      Intel C600 Series - SATA

                      For the verbose boot I will need to wait for them to provision the ILOM IP details on the network. Unfortunately I cannot access the console locally.

                      Reinaldo, would you be able to give me a hint as to how? It seems that the network interfaces are not managed by this tool?

                      cfgadm -la
                      Ap_Id Type Receptacle Occupant Condition
                      c0 scsi-sas connected configured unknown
                      c0::dsk/c0t3C77E758DC221474d0 disk connected configured unknown
                      c3 scsi-sas connected unconfigured unknown
                      c4 scsi-sas connected unconfigured unknown
                      sata0/0::dsk/c2t0d0 cd/dvd connected configured ok

                      Besides these devices only a number of usb devices are detected.
                      • 8. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                        800381
                        Yeah, the logic for putting scanpci in the X11 package is lost on me. It's a pretty basic and very useful tool. If you can just copy the binary from an exising Solaris box that has X11 installed, along with the /usr/X11/lib/modules/libscanpci.so shared object that the binary needs, you should be able to run it.

                        Although since the NICs are showing up in prtdiag, they'd almost certainly show up in scanpci output.

                        The verbose boot output is probably your best bet.

                        Another thing to check is the server BIOS settings - perhaps they're disabled in BIOS?
                        • 9. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
                          997789
                          Guys, thanks for your help!

                          I have finally located the issue, by searching through all system files for the 'ixgbe' string. The issue is located within the snapshot_cache package: http://docs.oracle.com/cd/E19082-01/819-2251/6n4i7tdd8/index.html

                          The idea of this cache is to remember/persist device identifiers across reboots. It consists of a folder with binary files, of which the documentation says: "There are no administrative actions necessary with respect to files in +/etc/devices+". Right!

                          The solution here was to delete the cache files. They get re-created after a reconfigure reboot (reboot -- -r) and afterwards they actually DO contain the right pci devices. Since the PCI devices are now properly detected, the correct links in /dev/ are also created when devfsadm -v is used.

                          Why the reconfigure reboot does not update this cache is beyond me. In fact I would go as far as marking this as a bug: Hardware was changed, but the cache was not re-built.

                          Anyway, hopefully this information is of use to somebody in the future. Thanks for the help everyone!