IPMP failures on bge Interface

807559
    We've been testing IPMP on Solaris Sparc hosts that also have the Apani IPSec Agent installed. It works fine on older hosts that have 'qfe' and 'le' interfaces, but our v210's and T1000's with 'bge' interfaces have a problem. If we configure an IPMP group to use, say, bge0 and bge1 (with bge0 as the primary interface), it works fine. Disconnecting bge0 causes a failover to bge1, also fine. Disconnecting bge1 causes the following errors:
    -------
    Nov 2 10:32:29 cs22 in.mpathd[146]: NIC failure detected on bge1 of group test
    Nov 2 10:32:29 cs22 in.mpathd[146]: Successfully failed over from NIC bge1 to NIC bge0
    Nov 2 10:32:37 cs2 in.mpathd[146]: All Interfaces in group test have failed
    -------
    All interfaces fail, even though bge0 is still connected and was active before disconnecting bge1. The system recovers once bge0 is reconnected. The two interfaces are physically connected to the same switch, and the hostname.bgeX files are:

    -------- hostname.bge0
    cs22 netmask + broadcast + group test up \
    addif cs21 deprecated -failover netmask + broadcast + up

    -------- hostname.bge1
    sp12 netmask + broadcast + group test up \
    addif sp16 deprecated -failover netmask + broadcast + up

    Any help would be appreciated, thanks in advance.
      • 1. Re: IPMP failures on bge Interface
        807559
        Could you post:
        + showrev
        + netstat -nr
        + /etc/hosts file
        + ifconfig -a (when bge0 and bge1 are connected)
        + ifconfig -a (after removing bge1)
        + ifconfig -a (after inserting bge1)
        + ifconfig -a (after removing bge0)
        + ifconfig -a (after inserting bge0)
        + /var/adm/messages file
        • 2. Re: IPMP failures on bge Interface
          807559
          Thanks for replying. Here's the requested information:
          -> showrev
          Hostname: cstoc77022
          Hostid: 842a9b82
          Release: 5.10
          Kernel architecture: sun4v
          Application architecture: sparc
          Hardware provider: Sun_Microsystems
          Domain: nis.nl.com
          Kernel version: SunOS 5.10 Generic_118833-03
          
          -> netstat -rn
          
          Routing Table: IPv4
            Destination           Gateway           Flags  Ref   Use   Interface
          -------------------- -------------------- ----- ----- ------ ---------
          63.192.85.64         63.192.77.9          UG        1      0  bge0
          63.192.78.0          63.192.77.9          UG        1      0  bge0
          63.192.77.0          63.192.77.22         U         1    162  bge0
          63.192.77.0          63.192.77.12         U         1     12  bge1
          63.192.77.0          63.192.77.12         U         1      0  bge0:1
          63.192.77.0          63.192.77.12         U         1      0  bge1:1
          63.192.76.0          63.192.77.9          UG        1      0  bge0
          10.3.0.0             63.192.77.92         UG        1      0  bge0
          172.20.0.0           63.192.77.4          UG        1      0  bge0
          172.16.0.0           63.192.77.9          UG        1      0  bge0
          10.0.0.0             63.192.77.9          UG        1      0  bge0
          224.0.0.0            63.192.77.22         U         1      0  bge0
          127.0.0.1            127.0.0.1            UH        7    328  lo0
          
          -> more /etc/hosts
          #
          # Internet host table
          #
          127.0.0.1       localhost       
          63.192.77.22    cstoc77022      loghost
          63.192.77.1     mls1    
          
          ---- BOTH CONNECTED: bge0, bge1
          -> ifconfig -a
          lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                  inet 127.0.0.1 netmask ff000000 
          bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                  inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:82 
          bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                  inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
          bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                  inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:83 
          bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                  inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
          
          
          ---- REMOVING bge1
          lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                  inet 127.0.0.1 netmask ff000000
          bge0: flags=1011000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FAILED,FIXEDMTU> mtu 1442 index 2
                  inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:82
          bge0:1: flags=19040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 i
          ndex 2
                  inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
          bge0:2: flags=1011000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FAILED,FIXEDMTU> mtu 1442 index 2
                  inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
          bge1: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 3
                  inet 0.0.0.0 netmask 0
                  groupname test
                  ether 0:14:4f:2a:9b:83
          bge1:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 3
                  inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
          
          Nov  2 13:00:22 cstoc77022 bge: NOTICE: bge1: link down
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: The link has gone down on bge1
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: NIC failure detected on bge1 of group test
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: Successfully failed over from NIC bge1 to NIC bge0
          Nov  2 13:00:30 cstoc77022 in.mpathd[146]: All Interfaces in group test have failed
          
          ---- INSERTING bge1
          lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                  inet 127.0.0.1 netmask ff000000
          bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                  inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:82
          bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                  inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
          bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                  inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:83
          bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                  inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
          
          
          Nov  2 13:01:59 cstoc77022 bge: NOTICE: bge1: link up 100Mbps Full-Duplex
          Nov  2 13:01:59 cstoc77022 in.mpathd[146]: The link has come up on bge1
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: NIC repair detected on bge1 of group test
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge1
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: At least 1 interface (bge1) of group test has repaired
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: NIC repair detected on bge0 of group test
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge0
          
          ---- REMOVING bge0
          lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                  inet 127.0.0.1 netmask ff000000
          bge0: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 2
                  inet 0.0.0.0 netmask 0
                  groupname test
                  ether 0:14:4f:2a:9b:82
          bge0:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 2
                  inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
          bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                  inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:83
          bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                  inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
          bge1:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                  inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
          
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: The link has gone down on bge0
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: NIC failure detected on bge0 of group test
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: Successfully failed over from NIC bge0 to NIC bge1
          
          ---- INSERTING bge0
          lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                  inet 127.0.0.1 netmask ff000000 
          bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                  inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:82 
          bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                  inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
          bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                  inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                  groupname test
                  ether 0:14:4f:2a:9b:83 
          bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                  inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
          
          Nov  2 13:04:20 cstoc77022 bge: NOTICE: bge0: link up 100Mbps Full-Duplex
          Nov  2 13:04:20 cstoc77022 in.mpathd[146]: The link has come up on bge0
          Nov  2 13:04:34 cstoc77022 in.mpathd[146]: NIC repair detected on bge0 of group test
          Nov  2 13:04:34 cstoc77022 ip: WARNING: IP: Proxy ARP problem?  Hardware address '00:14:4f:2a:9b:82
          ' thinks it is 063.192.077.022
          Nov  2 13:04:34 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge0
          
          ------ /var/adm/messages
          Nov  2 12:55:54 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
          Nov  2 12:57:23 cstoc77022 last message repeated 5 times
          Nov  2 13:00:22 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge1: link down
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 215189 daemon.error] The link has gone down on bge1
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 594170 daemon.error] NIC failure detected on bge1 of group test
          Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 832587 daemon.error] Successfully failed over from NIC bge1 to NIC bge0
          Nov  2 13:00:30 cstoc77022 in.mpathd[146]: [ID 168056 daemon.error] All Interfaces in group test have failed
          Nov  2 13:01:59 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge1: link up 100Mbps Full-Duplex
          Nov  2 13:01:59 cstoc77022 in.mpathd[146]: [ID 820239 daemon.error] The link has come up on bge1
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge1 of group test
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge1
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 237757 daemon.error] At least 1 interface (bge1) of group test has repaired
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge0 of group test
          Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge0
          Nov  2 13:02:55 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
          Nov  2 13:02:55 cstoc77022 last message repeated 1 time
          Nov  2 13:03:20 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge0: link down
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 215189 daemon.error] The link has gone down on bge0
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 594170 daemon.error] NIC failure detected on bge0 of group test
          Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 832587 daemon.error] Successfully failed over from NIC bge0 to NIC bge1
          Nov  2 13:04:20 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge0: link up 100Mbps Full-Duplex
          Nov  2 13:04:20 cstoc77022 in.mpathd[146]: [ID 820239 daemon.error] The link has come up on bge0
          Nov  2 13:04:34 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge0 of group test
          Nov  2 13:04:34 cstoc77022 ip: [ID 388441 kern.warning] WARNING: IP: Proxy ARP problem?  Hardware address '00:14:4f:2a:9b:82' thinks it is 063.192.077.022
          Nov  2 13:04:34 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge0
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 10.0.0.0 --> 63.192.77.9 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.78.0/24 --> 63.192.77.9 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.85.64/27 --> 63.192.77.9 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 172.20.0.0 --> 63.192.77.4 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 10.3.0.0/16 --> 63.192.77.92 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 172.16.0.0 --> 63.192.77.9 disappeared from kernel
          Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.76.0/24 --> 63.192.77.9 disappeared from kernel
          Nov  2 13:05:31 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
          • 3. Re: IPMP failures on bge Interface
            807559
            Could you post:
            + routeadm
            + arp -an
            + ps -aef
            + /etc/defaultrouter file
            Check your box is not acting as a router.
            Your box does not have a default router. Is that configuration right?
            The documentation states:
            << Routers that are connected to the IP link are automatically selected as targets for probing. If no routers exist on the link, in.mpathd sends probes to neighbor hosts on the link. A multicast packet that is sent to the all host multicast address.
            ...snip...
            If in.mpathd cannot find routers or hosts that responded to the ICMP echo packets, in.mpathd cannot detect probe-based failures.>>

            Is it allowed to send ICMP echo packet to the routers in the other networks?
            • 4. Re: IPMP failures on bge Interface
              807559
              Hello again,

              When gathering data for the previous reply, I also noticed that the default route had not been set. We usually do specify that, so I added that to the configuration. But, the host had found the correct router previously, it's 63.192.77.9. Specifying it did not change the problem symptoms, anyway. Here's the other requested info:
              -> netstat -rn
              
              Routing Table: IPv4
                Destination           Gateway           Flags  Ref   Use   Interface
              -------------------- -------------------- ----- ----- ------ ---------
              63.192.77.0          63.192.77.12         U         1      5  bge1
              63.192.77.0          63.192.77.22         U         1      1  bge0
              63.192.77.0          63.192.77.22         U         1      0  bge0:1
              63.192.77.0          63.192.77.12         U         1      0  bge1:1
              224.0.0.0            63.192.77.22         U         1      0  bge0
              default              63.192.77.9          UG        1      0  
              127.0.0.1            127.0.0.1            UH        7     93  lo0
              
              -> routeadm
                            Configuration   Current              Current
                                   Option   Configuration        System State
              ---------------------------------------------------------------
                          IPv4 forwarding   disabled             disabled
                             IPv4 routing   default (disabled)   disabled
                          IPv6 forwarding   disabled             disabled
                             IPv6 routing   disabled             disabled
              
                      IPv4 routing daemon   "/usr/sbin/in.routed"
                 IPv4 routing daemon args   ""
                 IPv4 routing daemon stop   "kill -TERM `cat /var/tmp/in.routed.pid`"
                      IPv6 routing daemon   "/usr/lib/inet/in.ripngd"
                 IPv6 routing daemon args   "-s"
                 IPv6 routing daemon stop   "kill -TERM `cat /var/tmp/in.ripngd.pid`"
              r
              -> arp -an
              Net to Media Table: IPv4
              Device   IP Address               Mask      Flags   Phys Addr 
              ------ -------------------- --------------- ----- ---------------
              bge1   63.192.77.1          255.255.255.255       00:03:ba:c0:77:75
              bge0   63.192.77.9          255.255.255.255       00:16:46:f1:b5:c2
              bge1   63.192.77.9          255.255.255.255       00:16:46:f1:b5:c2
              bge1   63.192.77.186        255.255.255.255       00:c0:4f:60:6a:ab
              bge0   63.192.77.186        255.255.255.255       00:c0:4f:60:6a:ab
              bge1   63.192.77.191        255.255.255.255       00:0c:f1:bf:1d:01
              bge0   63.192.77.191        255.255.255.255       00:0c:f1:bf:1d:01
              bge1   63.192.77.169        255.255.255.255       00:0c:f1:bf:1c:92
              bge0   63.192.77.169        255.255.255.255       00:0c:f1:bf:1c:92
              bge1   63.192.77.175        255.255.255.255       00:c0:4f:60:68:64
              bge0   63.192.77.175        255.255.255.255       00:c0:4f:60:68:64
              bge1   63.192.77.144        255.255.255.255       00:c0:4f:60:68:94
              bge0   63.192.77.144        255.255.255.255       00:c0:4f:60:68:94
              bge1   63.192.77.150        255.255.255.255       00:c0:4f:60:6a:70
              bge0   63.192.77.150        255.255.255.255       00:c0:4f:60:6a:70
              bge0   63.192.77.130        255.255.255.255       00:0c:f1:bf:1d:1f
              bge1   63.192.77.130        255.255.255.255       00:0c:f1:bf:1d:1f
              bge1   63.192.77.128        255.255.255.255       00:0c:f1:bf:1c:65
              bge0   63.192.77.128        255.255.255.255       00:0c:f1:bf:1c:65
              bge1   63.192.77.242        255.255.255.255       00:0d:56:0b:eb:2a
              bge0   63.192.77.242        255.255.255.255       00:0d:56:0b:eb:2a
              bge1   63.192.77.243        255.255.255.255       00:0f:1f:91:c1:9b
              bge0   63.192.77.243        255.255.255.255       00:0f:1f:91:c1:9b
              bge1   63.192.77.240        255.255.255.255       00:13:72:17:cb:13
              bge0   63.192.77.240        255.255.255.255       00:13:72:17:cb:13
              bge1   63.192.77.247        255.255.255.255       00:c0:4f:60:6a:e6
              bge0   63.192.77.247        255.255.255.255       00:c0:4f:60:6a:e6
              bge1   63.192.77.224        255.255.255.255       00:09:6b:2e:61:dd
              bge0   63.192.77.224        255.255.255.255       00:09:6b:2e:61:dd
              bge1   63.192.77.225        255.255.255.255       00:11:11:c4:9c:eb
              bge0   63.192.77.225        255.255.255.255       00:11:11:c4:9c:eb
              bge1   63.192.77.236        255.255.255.255       00:03:ba:eb:17:6d
              bge0   63.192.77.236        255.255.255.255       00:03:ba:eb:17:6d
              bge1   63.192.77.210        255.255.255.255       00:11:11:b1:2b:6e
              bge0   63.192.77.210        255.255.255.255       00:11:11:b1:2b:6e
              bge1   63.192.77.222        255.255.255.255       00:30:6e:08:ed:3a
              bge0   63.192.77.222        255.255.255.255       00:30:6e:08:ed:3a
              bge1   63.192.77.193        255.255.255.255       00:13:72:23:32:aa
              bge0   63.192.77.193        255.255.255.255       00:13:72:23:32:aa
              bge1   63.192.77.207        255.255.255.255       00:0c:f1:b6:26:aa
              bge0   63.192.77.207        255.255.255.255       00:0c:f1:b6:26:aa
              bge1   63.192.77.204        255.255.255.255       00:c0:4f:60:68:5b
              bge0   63.192.77.204        255.255.255.255       00:c0:4f:60:68:5b
              bge1   63.192.77.48         255.255.255.255       00:0a:95:99:e4:40
              bge0   63.192.77.48         255.255.255.255       00:0a:95:99:e4:40
              bge0   63.192.77.49         255.255.255.255       00:03:93:90:52:f6
              bge1   63.192.77.61         255.255.255.255       00:c0:4f:60:6a:75
              bge0   63.192.77.61         255.255.255.255       00:c0:4f:60:6a:75
              bge1   63.192.77.35         255.255.255.255       00:30:6e:49:41:50
              bge0   63.192.77.35         255.255.255.255       00:30:6e:49:41:50
              bge1   63.192.77.36         255.255.255.255       00:16:35:3e:7d:0a
              bge0   63.192.77.36         255.255.255.255       00:16:35:3e:7d:0a
              bge0   63.192.77.42         255.255.255.255       00:11:11:c4:9d:05
              bge1   63.192.77.42         255.255.255.255       00:11:11:c4:9d:05
              bge1   63.192.77.40         255.255.255.255       00:0c:f1:bf:1f:8d
              bge0   63.192.77.40         255.255.255.255       00:0c:f1:bf:1f:8d
              bge1   63.192.77.41         255.255.255.255       00:0c:f1:bf:1d:10
              bge0   63.192.77.41         255.255.255.255       00:0c:f1:bf:1d:10
              bge0   63.192.77.19         255.255.255.255       08:00:20:f0:ea:e4
              bge1   63.192.77.19         255.255.255.255       08:00:20:f0:ea:e4
              bge1   63.192.77.16         255.255.255.255 SP    00:14:4f:2a:9b:83
              bge0   63.192.77.22         255.255.255.255 SP    00:14:4f:2a:9b:82
              bge0   63.192.77.23         255.255.255.255       00:09:6b:3e:2b:82
              bge1   63.192.77.23         255.255.255.255       00:09:6b:3e:2b:82
              bge0   63.192.77.21         255.255.255.255 SP    00:14:4f:2a:9b:82
              bge1   63.192.77.29         255.255.255.255       00:09:6b:2e:46:51
              bge0   63.192.77.29         255.255.255.255       00:09:6b:2e:46:51
              bge0   63.192.77.1          255.255.255.255       00:03:ba:c0:77:75
              bge1   63.192.77.12         255.255.255.255 SP    00:14:4f:2a:9b:83
              bge0   63.192.77.115        255.255.255.255       00:0c:f1:bf:1c:e6
              bge1   63.192.77.115        255.255.255.255       00:0c:f1:bf:1c:e6
              bge1   63.192.77.122        255.255.255.255       00:10:83:f9:34:d4
              bge0   63.192.77.122        255.255.255.255       00:10:83:f9:34:d4
              bge1   63.192.77.125        255.255.255.255       00:0f:1f:91:bf:7d
              bge0   63.192.77.125        255.255.255.255       00:0f:1f:91:bf:7d
              bge1   63.192.77.99         255.255.255.255       00:0c:f1:bf:1a:52
              bge0   63.192.77.99         255.255.255.255       00:0c:f1:bf:1a:52
              bge1   63.192.77.100        255.255.255.255       00:0c:f1:b6:26:b4
              bge0   63.192.77.100        255.255.255.255       00:0c:f1:b6:26:b4
              bge1   63.192.77.101        255.255.255.255       00:0c:f1:bf:1c:fe
              bge0   63.192.77.101        255.255.255.255       00:0c:f1:bf:1c:fe
              bge1   63.192.77.107        255.255.255.255       00:0d:56:14:48:4d
              bge0   63.192.77.107        255.255.255.255       00:0d:56:14:48:4d
              bge1   63.192.77.110        255.255.255.255       00:c0:4f:60:6a:44
              bge0   63.192.77.110        255.255.255.255       00:c0:4f:60:6a:44
              bge1   63.192.77.108        255.255.255.255       00:14:bf:31:ec:e2
              bge0   63.192.77.108        255.255.255.255       00:14:bf:31:ec:e2
              bge0   63.192.77.80         255.255.255.255       00:16:cb:a6:5e:3d
              bge1   63.192.77.80         255.255.255.255       00:16:cb:a6:5e:3d
              bge1   63.192.77.92         255.255.255.255       00:40:63:d3:8c:46
              bge0   63.192.77.92         255.255.255.255       00:40:63:d3:8c:46
              bge1   63.192.77.68         255.255.255.255       00:0c:f1:b6:27:10
              bge0   63.192.77.68         255.255.255.255       00:0c:f1:b6:27:10
              bge1   63.192.77.69         255.255.255.255       00:13:72:17:ca:4a
              bge0   63.192.77.69         255.255.255.255       00:13:72:17:ca:4a
              bge1   63.192.77.73         255.255.255.255       00:03:93:d1:db:cc
              bge0   63.192.77.73         255.255.255.255       00:03:93:d1:db:cc
              bge1   63.192.77.77         255.255.255.255       00:30:65:a8:22:bc
              bge0   63.192.77.77         255.255.255.255       00:30:65:a8:22:bc
              bge1   224.0.0.0            240.0.0.0       SM    01:00:5e:00:00:00
              bge0   224.0.0.0            240.0.0.0       SM    01:00:5e:00:00:00
              
              -> ps -aef
                   UID   PID  PPID   C    STIME TTY         TIME CMD
                  root     0     0   0 15:11:12 ?           0:11 sched
                  root     1     0   0 15:11:13 ?           0:00 /sbin/init
                  root     2     0   0 15:11:13 ?           0:00 pageout
                  root     3     0   0 15:11:13 ?           0:00 fsflush
                daemon   196     1   0 15:11:37 ?           0:00 /usr/sbin/rpcbind
                  root     7     1   0 15:11:15 ?           0:10 /lib/svc/bin/svc.startd
                  root     9     1   0 15:11:16 ?           0:16 /lib/svc/bin/svc.configd
                  root   256     1   0 15:11:40 ?           0:00 /usr/sbin/cron
                  root   335     1   0 15:11:49 ?           0:00 /usr/sbin/syslogd
                  root   113     1   0 15:11:33 ?           0:00 /usr/sbin/nscd -S passwd,yes
                  root   726   691   0 15:16:16 pts/1       0:00 ps -aef
                daemon   201     1   0 15:11:37 ?           0:00 /usr/lib/nfs/statd
                  root   200     1   0 15:11:37 ?           0:00 /usr/sbin/keyserv
                  root   192     1   0 15:11:36 ?           0:01 /opt/apani/uagent/nlagent
                daemon    86     1   0 15:11:26 ?           0:00 /usr/lib/crypto/kcfd
                  root   152     1   0 15:11:35 ?           0:00 /usr/lib/inet/in.mpathd -a
                  root   212     7   0 15:11:38 ?           0:00 /usr/lib/saf/sac -t 300
                  root    89     1   0 15:11:26 ?           0:00 /usr/lib/picl/picld
                daemon   247     1   0 15:11:40 ?           0:00 /usr/lib/nfs/nfs4cbd
                  root   102     1   0 15:11:28 ?           0:00 /usr/lib/power/powerd
                  root    98     1   0 15:11:27 ?           0:00 /usr/lib/sysevent/syseventd
                  root   215     1   0 15:11:38 ?           0:00 /usr/sbin/nis_cachemgr
                daemon   214     1   0 15:11:38 ?           0:00 /usr/lib/nfs/lockd
                  root   213     1   0 15:11:38 ?           0:00 /usr/lib/utmpd
                  root   217     7   0 15:11:38 console     0:00 -sh
                  root   223   192   0 15:11:39 ?           0:00 inm -p9165
                  root   222   212   0 15:11:39 ?           0:00 /usr/lib/saf/ttymon
                daemon   255     1   0 15:11:40 ?           0:00 /usr/lib/nfs/nfsmapid
                  root   399   397   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                  root   252     1   0 15:11:40 ?           0:04 /usr/lib/inet/inetd start
                  root   398   397   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                  root   317     1   0 15:11:48 ?           0:00 /usr/lib/autofs/automountd
                  root   359     1   0 15:11:50 ?           0:00 /usr/lib/sendmail -bd -q15m
                  root   448   447   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/jserver_m
                  root   351     1   0 15:11:50 ?           0:02 /usr/lib/fm/fmd/fmd
                  root   674   252   0 15:12:14 ?           0:00 /usr/sbin/in.telnetd
                  root   347     1   0 15:11:50 ?           0:00 /usr/lib/ssh/sshd
                 smmsp   360     1   0 15:11:50 ?           0:00 /usr/lib/sendmail -Ac -q15m
                  root   461     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/atokserver/atokmngdaemon
                  root   397     1   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                  root   468   459   0 15:11:53 ?           0:00 htt_server -port 9010 -syslog -message_locale C
                  root   441     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/dpkeyserv
                  root   447     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/jserver
                  root   459     1   0 15:11:53 ?           0:00 /usr/lib/im/htt -port 9010 -syslog -message_locale C
                  root   512     1   0 15:11:55 ?           0:00 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
                  root   520     1   0 15:11:56 ?           0:00 /usr/lib/dmi/dmispd
                  root   528     1   0 15:11:56 ?           0:00 /usr/sbin/vold
                  root   521     1   0 15:11:56 ?           0:00 /usr/lib/dmi/snmpXdmid -s cstoc77022
                  root   511     1   0 15:11:55 ?           0:00 /usr/dt/bin/dtlogin -daemon
                  root   691   677   0 15:12:18 pts/1       0:00 bash
                  root   677   674   0 15:12:14 pts/1       0:00 -sh
                  root   585     1   0 15:11:57 ?           0:00 /usr/sfw/sbin/snmpd
              • 5. Re: IPMP failures on bge Interface
                807559
                Be sure your changes are permanent, reboot the box and try your tests again. IPMP finds targets at boot time.
                If it does not work, post all information as requested in reply 1 and 3.
                • 6. Re: IPMP failures on bge Interface
                  807559
                  OK. Actually, I had permanized and rebooted before the previous reply, but I had not rechecked all the ifconfig settings. Here there are again, this time with a configured default router:
                  -> netstat -rn
                  
                  Routing Table: IPv4
                    Destination           Gateway           Flags  Ref   Use   Interface
                  -------------------- -------------------- ----- ----- ------ ---------
                  63.192.77.0          63.192.77.22         U         1     21  bge0
                  63.192.77.0          63.192.77.12         U         1      1  bge1
                  63.192.77.0          63.192.77.22         U         1      0  bge0:1
                  63.192.77.0          63.192.77.22         U         1      0  bge1:1
                  224.0.0.0            63.192.77.22         U         1      0  bge0
                  default              63.192.77.9          UG        1      1  
                  127.0.0.1            127.0.0.1            UH        7     99  lo0
                  
                  ---- BOTH CONNECTED bge0, bge1
                  -> ifconfig -a
                  lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                          inet 127.0.0.1 netmask ff000000 
                  bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                          inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:82 
                  bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                          inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                  bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                          inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:83 
                  bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                          inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                  
                  ---- REMOVED bge1
                  -> ifconfig -a
                  lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                          inet 127.0.0.1 netmask ff000000 
                  bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                          inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:82 
                  bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                          inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                  bge0:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                          inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                  bge1: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 3
                          inet 0.0.0.0 netmask 0 
                          groupname test
                          ether 0:14:4f:2a:9b:83 
                  bge1:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 3
                          inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                  
                  Nov  2 16:47:59 cstoc77022 bge: NOTICE: bge1: link down
                  Nov  2 16:47:59 cstoc77022 in.mpathd[153]: The link has gone down on bge1
                  Nov  2 16:47:59 cstoc77022 in.mpathd[153]: NIC failure detected on bge1 of group test
                  Nov  2 16:47:59 cstoc77022 in.mpathd[153]: Successfully failed over from NIC bge1 to NIC bge0
                  Nov  2 16:48:07 cstoc77022 in.mpathd[153]: All Interfaces in group test have failed
                  
                  ---- RESTORED bge1
                  -> ifconfig -a
                  lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                          inet 127.0.0.1 netmask ff000000 
                  bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                          inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:82 
                  bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                          inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                  bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                          inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:83 
                  bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                          inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                  
                  Nov  2 16:48:51 cstoc77022 bge: NOTICE: bge1: link up 100Mbps Full-Duplex
                  Nov  2 16:48:51 cstoc77022 in.mpathd[153]: The link has come up on bge1
                  Nov  2 16:49:06 cstoc77022 in.mpathd[153]: NIC repair detected on bge0 of group test
                  Nov  2 16:49:06 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge0
                  Nov  2 16:49:06 cstoc77022 in.mpathd[153]: At least 1 interface (bge0) of group test has repaired
                  Nov  2 16:49:06 cstoc77022 in.mpathd[153]: NIC repair detected on bge1 of group test
                  Nov  2 16:49:06 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge1
                  
                  ---- REMOVED bge0
                  -> ifconfig -a
                  lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                          inet 127.0.0.1 netmask ff000000 
                  bge0: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 2
                          inet 0.0.0.0 netmask 0 
                          groupname test
                          ether 0:14:4f:2a:9b:82 
                  bge0:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 2
                          inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                  bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                          inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:83 
                  bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                          inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                  bge1:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                          inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                  
                  Nov  2 16:50:02 cstoc77022 bge: NOTICE: bge0: link down
                  Nov  2 16:50:02 cstoc77022 in.mpathd[153]: The link has gone down on bge0
                  Nov  2 16:50:02 cstoc77022 in.mpathd[153]: NIC failure detected on bge0 of group test
                  Nov  2 16:50:02 cstoc77022 in.mpathd[153]: Successfully failed over from NIC bge0 to NIC bge1
                  
                  ---- RESTORED bge0
                  -> ifconfig -a
                  lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                          inet 127.0.0.1 netmask ff000000 
                  bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                          inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:82 
                  bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                          inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                  bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                          inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                          groupname test
                          ether 0:14:4f:2a:9b:83 
                  bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                          inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                  
                  Nov  2 16:51:12 cstoc77022 bge: NOTICE: bge0: link up 100Mbps Full-Duplex
                  Nov  2 16:51:12 cstoc77022 in.mpathd[153]: The link has come up on bge0
                  Nov  2 16:51:12 cstoc77022 ip: WARNING: IP: Hardware address '00:14:4f:2a:9b:82' trying to be our a
                  ddress 063.192.077.021!
                  Nov  2 16:51:26 cstoc77022 in.mpathd[153]: NIC repair detected on bge0 of group test
                  Nov  2 16:51:26 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge0
                  Nov  2 16:51:34 cstoc77022 ip: WARNING: IP: Hardware address '00:14:4f:2a:9b:82' trying to be our a
                  ddress 063.192.077.022!
                  • 7. Re: IPMP failures on bge Interface
                    807559
                    1. Test your default router:
                    ping 63.192.77.9
                    2. Test another Sun boxes 63.192.77.1 and 63.192.77.236 and 63.192.77.19:
                    ping 63.192.77.1 ; ping 63.192.77.236 ; ping 63.192.77.19
                    3. If it works, add static routes and in a boot script:
                    route add -host 63.192.77.1 63.192.77.1 -static
                    route add -host 63.192.77.236 63.192.77.236 -static
                    route add -host 63.192.77.19 63.192.77.19 -static
                    4. Try your tests again.
                    5. If it does not work, install Recommended patches and bge patch (122027-08).
                    By the way, did your software uses arp for publishing MAC-IP addresses?
                    • 8. Re: IPMP failures on bge Interface
                      807559
                      I don't see how the explicit routes will change the results, but it's worth a try. Our software doesn't do anything with ARPs. The only thing we do is reduce the MTU size to make room for all the ESP headers. We've only had problems with the 'bge' interface, which is the mystery to us. Thanks for your help so far!
                      • 9. Re: IPMP failures on bge Interface
                        807559
                        The static routes didn't help so I installed the bge patch. It made my host unbootable, and since I'm using a Try&Buy T1000 there's no optical drive or external SCSI port. leaving a net install as my only option. I'm currently creating a JumpStart server, so hopefully my host will be back up for more testing later today.
                        • 10. Re: IPMP failures on bge Interface
                          807559
                          My host is back online, with a newer version of Solaris 10. It already has versions of the BGE patch, so I reran the IPMP tests. It now works normally for me. We'll test the corresponding patch for Solaris 8 also.

                          Thanks!