This discussion is archived
9 Replies Latest reply: Mar 21, 2013 9:04 AM by 997789 RSS

[M3 x4170] Missing network interfaces after installation 2nd CPU

997789 Newbie
Currently Being Moderated
Hi,

I have an M3 x4170 Oracle machine with initially one CPU installed. Solaris has been installed and detects two (out of the four) network interfaces. This is correct as the other two interfaces are unusable due to the missing CPU.

We have now moved this installation to a x4170 system with TWO CPUs installed. The additional CPU shows up, but the network interfaces do not.

# prtconf -D | grep ixgb
pci108e,484d, instance #0 (driver name: ixgbe)
pci108e,484d, instance #1 (driver name: ixgbe)
pci108e,484d (driver name: ixgbe)
pci108e,484d (driver name: ixgbe)

I have tried various combinations of 'touch /reconfigure; init6' and 'reboot -- -r' to have the system detect the new interfaces. This does not work.

# grep ixgbe /etc/path_to_inst
"/pci@0,0/pci8086,3c04@2/pci108e,484d@0" 0 "ixgbe"
"/pci@0,0/pci8086,3c04@2/pci108e,484d@0,1" 1 "ixgbe"

The pci devices seem to not get detected somehow. The path_to_inst file is missing references to the second controller. (Which is connected to the second CPU).

Any ideas on how to resolve this issue? The whole reason for getting a two-cpu system is the additional 10Gbit interfaces..

Thanks!
  • 1. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    Reidod Explorer
    Currently Being Moderated
    Hi,

    Can you post the output of 'dladm show-dev' command?

    Kind regards,
    Rei
  • 2. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    997789 Newbie
    Currently Being Moderated
    Hi Reidod, thanks for your help!

    The dladm shows two interfaces only, unfortunately:
    dladm show-dev
    ixgbe0 link: up speed: 1000 Mbps duplex: full
    ixgbe1 link: up speed: 1000 Mbps duplex: full
    My impression is that during Solaris installation the system detected that only one CPU was installed, and has written a config file/setting somewhere to disable the third and fourth interface. Now that we have added the second CPU that file is not updated. Its the only conclusion I can come up with since the device reconfigure steps do not change the situation. But where this file could be...

    Edited by: 994786 on Mar 20, 2013 6:39 AM

    Edited by: 994786 on Mar 20, 2013 6:39 AM
  • 3. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    Reidod Explorer
    Currently Being Moderated
    Hi,

    I would suggest to do a reconfiguration boot (append -r at kernel command in GRUB) or give 'devfsadm -v'. After that try to plumb the interfaces:

    ifconfig ixgbe2 plumb
    ifconfig ixgbe3 plumb

    Check if the interfaces will be plumbed. The driver for the NIC is located under /kernel/drv directory.

    Regards,
    Rei
  • 4. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    997789 Newbie
    Currently Being Moderated
    Hi Reidod,

    The devfsadm -v, reboot -- -r or touch /reconfigure should all kind of do the same thing, redetect the device tree. However when I do the /etc/path_to_inst is even updated. And therefore the additional devices are not loaded.

    A driver issue would not be expected, as the first two interfaces use the exact same driver as the second two. Therefore the driver is loaded and working.

    Since the low level devices are not available in /devices nor /dev I cannot plumb the interfaces yet, unfortunately..
  • 5. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    800381 Explorer
    Currently Being Moderated
    What does a verbose boot (reboot -- -r -v) show?

    What's the output from /usr/X11/bin/scanpci?

    Do all four network interfaces show on either of those?
  • 6. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    Reidod Explorer
    Currently Being Moderated
    Can you try to unconfigure/configure the PCI card using cfgadm?


    Regards,
    Reinaldo
  • 7. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    997789 Newbie
    Currently Being Moderated
    Thanks for your replies :)

    user5287726, scanpci is not available on this server - I do not use X11 at all I believe. The next best thing, prtdiag, shows all controllers:

    ==== On-Board Devices ======
    Onboard Video
    X540 10GbE Controller
    X540 10GbE Controller
    X540 10GbE Controller
    X540 10GbE Controller
    Intel C600 Series - SAS
    Intel C600 Series - SATA

    For the verbose boot I will need to wait for them to provision the ILOM IP details on the network. Unfortunately I cannot access the console locally.

    Reinaldo, would you be able to give me a hint as to how? It seems that the network interfaces are not managed by this tool?

    cfgadm -la
    Ap_Id Type Receptacle Occupant Condition
    c0 scsi-sas connected configured unknown
    c0::dsk/c0t3C77E758DC221474d0 disk connected configured unknown
    c3 scsi-sas connected unconfigured unknown
    c4 scsi-sas connected unconfigured unknown
    sata0/0::dsk/c2t0d0 cd/dvd connected configured ok

    Besides these devices only a number of usb devices are detected.
  • 8. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    800381 Explorer
    Currently Being Moderated
    Yeah, the logic for putting scanpci in the X11 package is lost on me. It's a pretty basic and very useful tool. If you can just copy the binary from an exising Solaris box that has X11 installed, along with the /usr/X11/lib/modules/libscanpci.so shared object that the binary needs, you should be able to run it.

    Although since the NICs are showing up in prtdiag, they'd almost certainly show up in scanpci output.

    The verbose boot output is probably your best bet.

    Another thing to check is the server BIOS settings - perhaps they're disabled in BIOS?
  • 9. Re: [M3 x4170] Missing network interfaces after installation 2nd CPU
    997789 Newbie
    Currently Being Moderated
    Guys, thanks for your help!

    I have finally located the issue, by searching through all system files for the 'ixgbe' string. The issue is located within the snapshot_cache package: http://docs.oracle.com/cd/E19082-01/819-2251/6n4i7tdd8/index.html

    The idea of this cache is to remember/persist device identifiers across reboots. It consists of a folder with binary files, of which the documentation says: "There are no administrative actions necessary with respect to files in +/etc/devices+". Right!

    The solution here was to delete the cache files. They get re-created after a reconfigure reboot (reboot -- -r) and afterwards they actually DO contain the right pci devices. Since the PCI devices are now properly detected, the correct links in /dev/ are also created when devfsadm -v is used.

    Why the reconfigure reboot does not update this cache is beyond me. In fact I would go as far as marking this as a bug: Hardware was changed, but the cache was not re-built.

    Anyway, hopefully this information is of use to somebody in the future. Thanks for the help everyone!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points