10 Replies Latest reply on Nov 6, 2006 11:58 PM by 807559

    IPMP failures on bge Interface

    807559
      We've been testing IPMP on Solaris Sparc hosts that also have the Apani IPSec Agent installed. It works fine on older hosts that have 'qfe' and 'le' interfaces, but our v210's and T1000's with 'bge' interfaces have a problem. If we configure an IPMP group to use, say, bge0 and bge1 (with bge0 as the primary interface), it works fine. Disconnecting bge0 causes a failover to bge1, also fine. Disconnecting bge1 causes the following errors:
      -------
      Nov 2 10:32:29 cs22 in.mpathd[146]: NIC failure detected on bge1 of group test
      Nov 2 10:32:29 cs22 in.mpathd[146]: Successfully failed over from NIC bge1 to NIC bge0
      Nov 2 10:32:37 cs2 in.mpathd[146]: All Interfaces in group test have failed
      -------
      All interfaces fail, even though bge0 is still connected and was active before disconnecting bge1. The system recovers once bge0 is reconnected. The two interfaces are physically connected to the same switch, and the hostname.bgeX files are:

      -------- hostname.bge0
      cs22 netmask + broadcast + group test up \
      addif cs21 deprecated -failover netmask + broadcast + up

      -------- hostname.bge1
      sp12 netmask + broadcast + group test up \
      addif sp16 deprecated -failover netmask + broadcast + up

      Any help would be appreciated, thanks in advance.
        • 1. Re: IPMP failures on bge Interface
          807559
          Could you post:
          + showrev
          + netstat -nr
          + /etc/hosts file
          + ifconfig -a (when bge0 and bge1 are connected)
          + ifconfig -a (after removing bge1)
          + ifconfig -a (after inserting bge1)
          + ifconfig -a (after removing bge0)
          + ifconfig -a (after inserting bge0)
          + /var/adm/messages file
          • 2. Re: IPMP failures on bge Interface
            807559
            Thanks for replying. Here's the requested information:
            -> showrev
            Hostname: cstoc77022
            Hostid: 842a9b82
            Release: 5.10
            Kernel architecture: sun4v
            Application architecture: sparc
            Hardware provider: Sun_Microsystems
            Domain: nis.nl.com
            Kernel version: SunOS 5.10 Generic_118833-03
            
            -> netstat -rn
            
            Routing Table: IPv4
              Destination           Gateway           Flags  Ref   Use   Interface
            -------------------- -------------------- ----- ----- ------ ---------
            63.192.85.64         63.192.77.9          UG        1      0  bge0
            63.192.78.0          63.192.77.9          UG        1      0  bge0
            63.192.77.0          63.192.77.22         U         1    162  bge0
            63.192.77.0          63.192.77.12         U         1     12  bge1
            63.192.77.0          63.192.77.12         U         1      0  bge0:1
            63.192.77.0          63.192.77.12         U         1      0  bge1:1
            63.192.76.0          63.192.77.9          UG        1      0  bge0
            10.3.0.0             63.192.77.92         UG        1      0  bge0
            172.20.0.0           63.192.77.4          UG        1      0  bge0
            172.16.0.0           63.192.77.9          UG        1      0  bge0
            10.0.0.0             63.192.77.9          UG        1      0  bge0
            224.0.0.0            63.192.77.22         U         1      0  bge0
            127.0.0.1            127.0.0.1            UH        7    328  lo0
            
            -> more /etc/hosts
            #
            # Internet host table
            #
            127.0.0.1       localhost       
            63.192.77.22    cstoc77022      loghost
            63.192.77.1     mls1    
            
            ---- BOTH CONNECTED: bge0, bge1
            -> ifconfig -a
            lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                    inet 127.0.0.1 netmask ff000000 
            bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                    inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:82 
            bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                    inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
            bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                    inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:83 
            bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                    inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
            
            
            ---- REMOVING bge1
            lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                    inet 127.0.0.1 netmask ff000000
            bge0: flags=1011000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FAILED,FIXEDMTU> mtu 1442 index 2
                    inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:82
            bge0:1: flags=19040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 i
            ndex 2
                    inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
            bge0:2: flags=1011000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FAILED,FIXEDMTU> mtu 1442 index 2
                    inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
            bge1: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 3
                    inet 0.0.0.0 netmask 0
                    groupname test
                    ether 0:14:4f:2a:9b:83
            bge1:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 3
                    inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
            
            Nov  2 13:00:22 cstoc77022 bge: NOTICE: bge1: link down
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: The link has gone down on bge1
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: NIC failure detected on bge1 of group test
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: Successfully failed over from NIC bge1 to NIC bge0
            Nov  2 13:00:30 cstoc77022 in.mpathd[146]: All Interfaces in group test have failed
            
            ---- INSERTING bge1
            lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                    inet 127.0.0.1 netmask ff000000
            bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                    inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:82
            bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                    inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
            bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                    inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:83
            bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                    inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
            
            
            Nov  2 13:01:59 cstoc77022 bge: NOTICE: bge1: link up 100Mbps Full-Duplex
            Nov  2 13:01:59 cstoc77022 in.mpathd[146]: The link has come up on bge1
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: NIC repair detected on bge1 of group test
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge1
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: At least 1 interface (bge1) of group test has repaired
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: NIC repair detected on bge0 of group test
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge0
            
            ---- REMOVING bge0
            lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                    inet 127.0.0.1 netmask ff000000
            bge0: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 2
                    inet 0.0.0.0 netmask 0
                    groupname test
                    ether 0:14:4f:2a:9b:82
            bge0:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 2
                    inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
            bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                    inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:83
            bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                    inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
            bge1:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                    inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
            
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: The link has gone down on bge0
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: NIC failure detected on bge0 of group test
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: Successfully failed over from NIC bge0 to NIC bge1
            
            ---- INSERTING bge0
            lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                    inet 127.0.0.1 netmask ff000000 
            bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                    inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:82 
            bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                    inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
            bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                    inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                    groupname test
                    ether 0:14:4f:2a:9b:83 
            bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                    inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
            
            Nov  2 13:04:20 cstoc77022 bge: NOTICE: bge0: link up 100Mbps Full-Duplex
            Nov  2 13:04:20 cstoc77022 in.mpathd[146]: The link has come up on bge0
            Nov  2 13:04:34 cstoc77022 in.mpathd[146]: NIC repair detected on bge0 of group test
            Nov  2 13:04:34 cstoc77022 ip: WARNING: IP: Proxy ARP problem?  Hardware address '00:14:4f:2a:9b:82
            ' thinks it is 063.192.077.022
            Nov  2 13:04:34 cstoc77022 in.mpathd[146]: Successfully failed back to NIC bge0
            
            ------ /var/adm/messages
            Nov  2 12:55:54 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
            Nov  2 12:57:23 cstoc77022 last message repeated 5 times
            Nov  2 13:00:22 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge1: link down
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 215189 daemon.error] The link has gone down on bge1
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 594170 daemon.error] NIC failure detected on bge1 of group test
            Nov  2 13:00:22 cstoc77022 in.mpathd[146]: [ID 832587 daemon.error] Successfully failed over from NIC bge1 to NIC bge0
            Nov  2 13:00:30 cstoc77022 in.mpathd[146]: [ID 168056 daemon.error] All Interfaces in group test have failed
            Nov  2 13:01:59 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge1: link up 100Mbps Full-Duplex
            Nov  2 13:01:59 cstoc77022 in.mpathd[146]: [ID 820239 daemon.error] The link has come up on bge1
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge1 of group test
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge1
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 237757 daemon.error] At least 1 interface (bge1) of group test has repaired
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge0 of group test
            Nov  2 13:02:14 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge0
            Nov  2 13:02:55 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
            Nov  2 13:02:55 cstoc77022 last message repeated 1 time
            Nov  2 13:03:20 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge0: link down
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 215189 daemon.error] The link has gone down on bge0
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 594170 daemon.error] NIC failure detected on bge0 of group test
            Nov  2 13:03:20 cstoc77022 in.mpathd[146]: [ID 832587 daemon.error] Successfully failed over from NIC bge0 to NIC bge1
            Nov  2 13:04:20 cstoc77022 bge: [ID 801593 kern.notice] NOTICE: bge0: link up 100Mbps Full-Duplex
            Nov  2 13:04:20 cstoc77022 in.mpathd[146]: [ID 820239 daemon.error] The link has come up on bge0
            Nov  2 13:04:34 cstoc77022 in.mpathd[146]: [ID 299542 daemon.error] NIC repair detected on bge0 of group test
            Nov  2 13:04:34 cstoc77022 ip: [ID 388441 kern.warning] WARNING: IP: Proxy ARP problem?  Hardware address '00:14:4f:2a:9b:82' thinks it is 063.192.077.022
            Nov  2 13:04:34 cstoc77022 in.mpathd[146]: [ID 620804 daemon.error] Successfully failed back to NIC bge0
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 10.0.0.0 --> 63.192.77.9 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.78.0/24 --> 63.192.77.9 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.85.64/27 --> 63.192.77.9 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 172.20.0.0 --> 63.192.77.4 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 10.3.0.0/16 --> 63.192.77.92 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 172.16.0.0 --> 63.192.77.9 disappeared from kernel
            Nov  2 13:04:44 cstoc77022 in.routed[158]: [ID 559541 daemon.warning] 63.192.76.0/24 --> 63.192.77.9 disappeared from kernel
            Nov  2 13:05:31 cstoc77022 nfs: [ID 664466 kern.notice] NFS getattr failed for server mls1: error 7 (RPC: Authentication error)
            • 3. Re: IPMP failures on bge Interface
              807559
              Could you post:
              + routeadm
              + arp -an
              + ps -aef
              + /etc/defaultrouter file
              Check your box is not acting as a router.
              Your box does not have a default router. Is that configuration right?
              The documentation states:
              << Routers that are connected to the IP link are automatically selected as targets for probing. If no routers exist on the link, in.mpathd sends probes to neighbor hosts on the link. A multicast packet that is sent to the all host multicast address.
              ...snip...
              If in.mpathd cannot find routers or hosts that responded to the ICMP echo packets, in.mpathd cannot detect probe-based failures.>>

              Is it allowed to send ICMP echo packet to the routers in the other networks?
              • 4. Re: IPMP failures on bge Interface
                807559
                Hello again,

                When gathering data for the previous reply, I also noticed that the default route had not been set. We usually do specify that, so I added that to the configuration. But, the host had found the correct router previously, it's 63.192.77.9. Specifying it did not change the problem symptoms, anyway. Here's the other requested info:
                -> netstat -rn
                
                Routing Table: IPv4
                  Destination           Gateway           Flags  Ref   Use   Interface
                -------------------- -------------------- ----- ----- ------ ---------
                63.192.77.0          63.192.77.12         U         1      5  bge1
                63.192.77.0          63.192.77.22         U         1      1  bge0
                63.192.77.0          63.192.77.22         U         1      0  bge0:1
                63.192.77.0          63.192.77.12         U         1      0  bge1:1
                224.0.0.0            63.192.77.22         U         1      0  bge0
                default              63.192.77.9          UG        1      0  
                127.0.0.1            127.0.0.1            UH        7     93  lo0
                
                -> routeadm
                              Configuration   Current              Current
                                     Option   Configuration        System State
                ---------------------------------------------------------------
                            IPv4 forwarding   disabled             disabled
                               IPv4 routing   default (disabled)   disabled
                            IPv6 forwarding   disabled             disabled
                               IPv6 routing   disabled             disabled
                
                        IPv4 routing daemon   "/usr/sbin/in.routed"
                   IPv4 routing daemon args   ""
                   IPv4 routing daemon stop   "kill -TERM `cat /var/tmp/in.routed.pid`"
                        IPv6 routing daemon   "/usr/lib/inet/in.ripngd"
                   IPv6 routing daemon args   "-s"
                   IPv6 routing daemon stop   "kill -TERM `cat /var/tmp/in.ripngd.pid`"
                r
                -> arp -an
                Net to Media Table: IPv4
                Device   IP Address               Mask      Flags   Phys Addr 
                ------ -------------------- --------------- ----- ---------------
                bge1   63.192.77.1          255.255.255.255       00:03:ba:c0:77:75
                bge0   63.192.77.9          255.255.255.255       00:16:46:f1:b5:c2
                bge1   63.192.77.9          255.255.255.255       00:16:46:f1:b5:c2
                bge1   63.192.77.186        255.255.255.255       00:c0:4f:60:6a:ab
                bge0   63.192.77.186        255.255.255.255       00:c0:4f:60:6a:ab
                bge1   63.192.77.191        255.255.255.255       00:0c:f1:bf:1d:01
                bge0   63.192.77.191        255.255.255.255       00:0c:f1:bf:1d:01
                bge1   63.192.77.169        255.255.255.255       00:0c:f1:bf:1c:92
                bge0   63.192.77.169        255.255.255.255       00:0c:f1:bf:1c:92
                bge1   63.192.77.175        255.255.255.255       00:c0:4f:60:68:64
                bge0   63.192.77.175        255.255.255.255       00:c0:4f:60:68:64
                bge1   63.192.77.144        255.255.255.255       00:c0:4f:60:68:94
                bge0   63.192.77.144        255.255.255.255       00:c0:4f:60:68:94
                bge1   63.192.77.150        255.255.255.255       00:c0:4f:60:6a:70
                bge0   63.192.77.150        255.255.255.255       00:c0:4f:60:6a:70
                bge0   63.192.77.130        255.255.255.255       00:0c:f1:bf:1d:1f
                bge1   63.192.77.130        255.255.255.255       00:0c:f1:bf:1d:1f
                bge1   63.192.77.128        255.255.255.255       00:0c:f1:bf:1c:65
                bge0   63.192.77.128        255.255.255.255       00:0c:f1:bf:1c:65
                bge1   63.192.77.242        255.255.255.255       00:0d:56:0b:eb:2a
                bge0   63.192.77.242        255.255.255.255       00:0d:56:0b:eb:2a
                bge1   63.192.77.243        255.255.255.255       00:0f:1f:91:c1:9b
                bge0   63.192.77.243        255.255.255.255       00:0f:1f:91:c1:9b
                bge1   63.192.77.240        255.255.255.255       00:13:72:17:cb:13
                bge0   63.192.77.240        255.255.255.255       00:13:72:17:cb:13
                bge1   63.192.77.247        255.255.255.255       00:c0:4f:60:6a:e6
                bge0   63.192.77.247        255.255.255.255       00:c0:4f:60:6a:e6
                bge1   63.192.77.224        255.255.255.255       00:09:6b:2e:61:dd
                bge0   63.192.77.224        255.255.255.255       00:09:6b:2e:61:dd
                bge1   63.192.77.225        255.255.255.255       00:11:11:c4:9c:eb
                bge0   63.192.77.225        255.255.255.255       00:11:11:c4:9c:eb
                bge1   63.192.77.236        255.255.255.255       00:03:ba:eb:17:6d
                bge0   63.192.77.236        255.255.255.255       00:03:ba:eb:17:6d
                bge1   63.192.77.210        255.255.255.255       00:11:11:b1:2b:6e
                bge0   63.192.77.210        255.255.255.255       00:11:11:b1:2b:6e
                bge1   63.192.77.222        255.255.255.255       00:30:6e:08:ed:3a
                bge0   63.192.77.222        255.255.255.255       00:30:6e:08:ed:3a
                bge1   63.192.77.193        255.255.255.255       00:13:72:23:32:aa
                bge0   63.192.77.193        255.255.255.255       00:13:72:23:32:aa
                bge1   63.192.77.207        255.255.255.255       00:0c:f1:b6:26:aa
                bge0   63.192.77.207        255.255.255.255       00:0c:f1:b6:26:aa
                bge1   63.192.77.204        255.255.255.255       00:c0:4f:60:68:5b
                bge0   63.192.77.204        255.255.255.255       00:c0:4f:60:68:5b
                bge1   63.192.77.48         255.255.255.255       00:0a:95:99:e4:40
                bge0   63.192.77.48         255.255.255.255       00:0a:95:99:e4:40
                bge0   63.192.77.49         255.255.255.255       00:03:93:90:52:f6
                bge1   63.192.77.61         255.255.255.255       00:c0:4f:60:6a:75
                bge0   63.192.77.61         255.255.255.255       00:c0:4f:60:6a:75
                bge1   63.192.77.35         255.255.255.255       00:30:6e:49:41:50
                bge0   63.192.77.35         255.255.255.255       00:30:6e:49:41:50
                bge1   63.192.77.36         255.255.255.255       00:16:35:3e:7d:0a
                bge0   63.192.77.36         255.255.255.255       00:16:35:3e:7d:0a
                bge0   63.192.77.42         255.255.255.255       00:11:11:c4:9d:05
                bge1   63.192.77.42         255.255.255.255       00:11:11:c4:9d:05
                bge1   63.192.77.40         255.255.255.255       00:0c:f1:bf:1f:8d
                bge0   63.192.77.40         255.255.255.255       00:0c:f1:bf:1f:8d
                bge1   63.192.77.41         255.255.255.255       00:0c:f1:bf:1d:10
                bge0   63.192.77.41         255.255.255.255       00:0c:f1:bf:1d:10
                bge0   63.192.77.19         255.255.255.255       08:00:20:f0:ea:e4
                bge1   63.192.77.19         255.255.255.255       08:00:20:f0:ea:e4
                bge1   63.192.77.16         255.255.255.255 SP    00:14:4f:2a:9b:83
                bge0   63.192.77.22         255.255.255.255 SP    00:14:4f:2a:9b:82
                bge0   63.192.77.23         255.255.255.255       00:09:6b:3e:2b:82
                bge1   63.192.77.23         255.255.255.255       00:09:6b:3e:2b:82
                bge0   63.192.77.21         255.255.255.255 SP    00:14:4f:2a:9b:82
                bge1   63.192.77.29         255.255.255.255       00:09:6b:2e:46:51
                bge0   63.192.77.29         255.255.255.255       00:09:6b:2e:46:51
                bge0   63.192.77.1          255.255.255.255       00:03:ba:c0:77:75
                bge1   63.192.77.12         255.255.255.255 SP    00:14:4f:2a:9b:83
                bge0   63.192.77.115        255.255.255.255       00:0c:f1:bf:1c:e6
                bge1   63.192.77.115        255.255.255.255       00:0c:f1:bf:1c:e6
                bge1   63.192.77.122        255.255.255.255       00:10:83:f9:34:d4
                bge0   63.192.77.122        255.255.255.255       00:10:83:f9:34:d4
                bge1   63.192.77.125        255.255.255.255       00:0f:1f:91:bf:7d
                bge0   63.192.77.125        255.255.255.255       00:0f:1f:91:bf:7d
                bge1   63.192.77.99         255.255.255.255       00:0c:f1:bf:1a:52
                bge0   63.192.77.99         255.255.255.255       00:0c:f1:bf:1a:52
                bge1   63.192.77.100        255.255.255.255       00:0c:f1:b6:26:b4
                bge0   63.192.77.100        255.255.255.255       00:0c:f1:b6:26:b4
                bge1   63.192.77.101        255.255.255.255       00:0c:f1:bf:1c:fe
                bge0   63.192.77.101        255.255.255.255       00:0c:f1:bf:1c:fe
                bge1   63.192.77.107        255.255.255.255       00:0d:56:14:48:4d
                bge0   63.192.77.107        255.255.255.255       00:0d:56:14:48:4d
                bge1   63.192.77.110        255.255.255.255       00:c0:4f:60:6a:44
                bge0   63.192.77.110        255.255.255.255       00:c0:4f:60:6a:44
                bge1   63.192.77.108        255.255.255.255       00:14:bf:31:ec:e2
                bge0   63.192.77.108        255.255.255.255       00:14:bf:31:ec:e2
                bge0   63.192.77.80         255.255.255.255       00:16:cb:a6:5e:3d
                bge1   63.192.77.80         255.255.255.255       00:16:cb:a6:5e:3d
                bge1   63.192.77.92         255.255.255.255       00:40:63:d3:8c:46
                bge0   63.192.77.92         255.255.255.255       00:40:63:d3:8c:46
                bge1   63.192.77.68         255.255.255.255       00:0c:f1:b6:27:10
                bge0   63.192.77.68         255.255.255.255       00:0c:f1:b6:27:10
                bge1   63.192.77.69         255.255.255.255       00:13:72:17:ca:4a
                bge0   63.192.77.69         255.255.255.255       00:13:72:17:ca:4a
                bge1   63.192.77.73         255.255.255.255       00:03:93:d1:db:cc
                bge0   63.192.77.73         255.255.255.255       00:03:93:d1:db:cc
                bge1   63.192.77.77         255.255.255.255       00:30:65:a8:22:bc
                bge0   63.192.77.77         255.255.255.255       00:30:65:a8:22:bc
                bge1   224.0.0.0            240.0.0.0       SM    01:00:5e:00:00:00
                bge0   224.0.0.0            240.0.0.0       SM    01:00:5e:00:00:00
                
                -> ps -aef
                     UID   PID  PPID   C    STIME TTY         TIME CMD
                    root     0     0   0 15:11:12 ?           0:11 sched
                    root     1     0   0 15:11:13 ?           0:00 /sbin/init
                    root     2     0   0 15:11:13 ?           0:00 pageout
                    root     3     0   0 15:11:13 ?           0:00 fsflush
                  daemon   196     1   0 15:11:37 ?           0:00 /usr/sbin/rpcbind
                    root     7     1   0 15:11:15 ?           0:10 /lib/svc/bin/svc.startd
                    root     9     1   0 15:11:16 ?           0:16 /lib/svc/bin/svc.configd
                    root   256     1   0 15:11:40 ?           0:00 /usr/sbin/cron
                    root   335     1   0 15:11:49 ?           0:00 /usr/sbin/syslogd
                    root   113     1   0 15:11:33 ?           0:00 /usr/sbin/nscd -S passwd,yes
                    root   726   691   0 15:16:16 pts/1       0:00 ps -aef
                  daemon   201     1   0 15:11:37 ?           0:00 /usr/lib/nfs/statd
                    root   200     1   0 15:11:37 ?           0:00 /usr/sbin/keyserv
                    root   192     1   0 15:11:36 ?           0:01 /opt/apani/uagent/nlagent
                  daemon    86     1   0 15:11:26 ?           0:00 /usr/lib/crypto/kcfd
                    root   152     1   0 15:11:35 ?           0:00 /usr/lib/inet/in.mpathd -a
                    root   212     7   0 15:11:38 ?           0:00 /usr/lib/saf/sac -t 300
                    root    89     1   0 15:11:26 ?           0:00 /usr/lib/picl/picld
                  daemon   247     1   0 15:11:40 ?           0:00 /usr/lib/nfs/nfs4cbd
                    root   102     1   0 15:11:28 ?           0:00 /usr/lib/power/powerd
                    root    98     1   0 15:11:27 ?           0:00 /usr/lib/sysevent/syseventd
                    root   215     1   0 15:11:38 ?           0:00 /usr/sbin/nis_cachemgr
                  daemon   214     1   0 15:11:38 ?           0:00 /usr/lib/nfs/lockd
                    root   213     1   0 15:11:38 ?           0:00 /usr/lib/utmpd
                    root   217     7   0 15:11:38 console     0:00 -sh
                    root   223   192   0 15:11:39 ?           0:00 inm -p9165
                    root   222   212   0 15:11:39 ?           0:00 /usr/lib/saf/ttymon
                  daemon   255     1   0 15:11:40 ?           0:00 /usr/lib/nfs/nfsmapid
                    root   399   397   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                    root   252     1   0 15:11:40 ?           0:04 /usr/lib/inet/inetd start
                    root   398   397   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                    root   317     1   0 15:11:48 ?           0:00 /usr/lib/autofs/automountd
                    root   359     1   0 15:11:50 ?           0:00 /usr/lib/sendmail -bd -q15m
                    root   448   447   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/jserver_m
                    root   351     1   0 15:11:50 ?           0:02 /usr/lib/fm/fmd/fmd
                    root   674   252   0 15:12:14 ?           0:00 /usr/sbin/in.telnetd
                    root   347     1   0 15:11:50 ?           0:00 /usr/lib/ssh/sshd
                   smmsp   360     1   0 15:11:50 ?           0:00 /usr/lib/sendmail -Ac -q15m
                    root   461     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/atokserver/atokmngdaemon
                    root   397     1   0 15:11:52 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
                    root   468   459   0 15:11:53 ?           0:00 htt_server -port 9010 -syslog -message_locale C
                    root   441     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/dpkeyserv
                    root   447     1   0 15:11:53 ?           0:00 /usr/lib/locale/ja/wnn/jserver
                    root   459     1   0 15:11:53 ?           0:00 /usr/lib/im/htt -port 9010 -syslog -message_locale C
                    root   512     1   0 15:11:55 ?           0:00 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
                    root   520     1   0 15:11:56 ?           0:00 /usr/lib/dmi/dmispd
                    root   528     1   0 15:11:56 ?           0:00 /usr/sbin/vold
                    root   521     1   0 15:11:56 ?           0:00 /usr/lib/dmi/snmpXdmid -s cstoc77022
                    root   511     1   0 15:11:55 ?           0:00 /usr/dt/bin/dtlogin -daemon
                    root   691   677   0 15:12:18 pts/1       0:00 bash
                    root   677   674   0 15:12:14 pts/1       0:00 -sh
                    root   585     1   0 15:11:57 ?           0:00 /usr/sfw/sbin/snmpd
                • 5. Re: IPMP failures on bge Interface
                  807559
                  Be sure your changes are permanent, reboot the box and try your tests again. IPMP finds targets at boot time.
                  If it does not work, post all information as requested in reply 1 and 3.
                  • 6. Re: IPMP failures on bge Interface
                    807559
                    OK. Actually, I had permanized and rebooted before the previous reply, but I had not rechecked all the ifconfig settings. Here there are again, this time with a configured default router:
                    -> netstat -rn
                    
                    Routing Table: IPv4
                      Destination           Gateway           Flags  Ref   Use   Interface
                    -------------------- -------------------- ----- ----- ------ ---------
                    63.192.77.0          63.192.77.22         U         1     21  bge0
                    63.192.77.0          63.192.77.12         U         1      1  bge1
                    63.192.77.0          63.192.77.22         U         1      0  bge0:1
                    63.192.77.0          63.192.77.22         U         1      0  bge1:1
                    224.0.0.0            63.192.77.22         U         1      0  bge0
                    default              63.192.77.9          UG        1      1  
                    127.0.0.1            127.0.0.1            UH        7     99  lo0
                    
                    ---- BOTH CONNECTED bge0, bge1
                    -> ifconfig -a
                    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                            inet 127.0.0.1 netmask ff000000 
                    bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                            inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:82 
                    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                            inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                    bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                            inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:83 
                    bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                            inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                    
                    ---- REMOVED bge1
                    -> ifconfig -a
                    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                            inet 127.0.0.1 netmask ff000000 
                    bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                            inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:82 
                    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                            inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                    bge0:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                            inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                    bge1: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 3
                            inet 0.0.0.0 netmask 0 
                            groupname test
                            ether 0:14:4f:2a:9b:83 
                    bge1:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 3
                            inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                    
                    Nov  2 16:47:59 cstoc77022 bge: NOTICE: bge1: link down
                    Nov  2 16:47:59 cstoc77022 in.mpathd[153]: The link has gone down on bge1
                    Nov  2 16:47:59 cstoc77022 in.mpathd[153]: NIC failure detected on bge1 of group test
                    Nov  2 16:47:59 cstoc77022 in.mpathd[153]: Successfully failed over from NIC bge1 to NIC bge0
                    Nov  2 16:48:07 cstoc77022 in.mpathd[153]: All Interfaces in group test have failed
                    
                    ---- RESTORED bge1
                    -> ifconfig -a
                    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                            inet 127.0.0.1 netmask ff000000 
                    bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                            inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:82 
                    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                            inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                    bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                            inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:83 
                    bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                            inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                    
                    Nov  2 16:48:51 cstoc77022 bge: NOTICE: bge1: link up 100Mbps Full-Duplex
                    Nov  2 16:48:51 cstoc77022 in.mpathd[153]: The link has come up on bge1
                    Nov  2 16:49:06 cstoc77022 in.mpathd[153]: NIC repair detected on bge0 of group test
                    Nov  2 16:49:06 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge0
                    Nov  2 16:49:06 cstoc77022 in.mpathd[153]: At least 1 interface (bge0) of group test has repaired
                    Nov  2 16:49:06 cstoc77022 in.mpathd[153]: NIC repair detected on bge1 of group test
                    Nov  2 16:49:06 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge1
                    
                    ---- REMOVED bge0
                    -> ifconfig -a
                    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                            inet 127.0.0.1 netmask ff000000 
                    bge0: flags=1019000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,FIXEDMTU> mtu 0 index 2
                            inet 0.0.0.0 netmask 0 
                            groupname test
                            ether 0:14:4f:2a:9b:82 
                    bge0:1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 2
                            inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                    bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                            inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:83 
                    bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                            inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                    bge1:2: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                            inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                    
                    Nov  2 16:50:02 cstoc77022 bge: NOTICE: bge0: link down
                    Nov  2 16:50:02 cstoc77022 in.mpathd[153]: The link has gone down on bge0
                    Nov  2 16:50:02 cstoc77022 in.mpathd[153]: NIC failure detected on bge0 of group test
                    Nov  2 16:50:02 cstoc77022 in.mpathd[153]: Successfully failed over from NIC bge0 to NIC bge1
                    
                    ---- RESTORED bge0
                    -> ifconfig -a
                    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                            inet 127.0.0.1 netmask ff000000 
                    bge0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 2
                            inet 63.192.77.22 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:82 
                    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
                            inet 63.192.77.21 netmask ffffff00 broadcast 63.192.77.255
                    bge1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1442 index 3
                            inet 63.192.77.12 netmask ffffff00 broadcast 63.192.77.255
                            groupname test
                            ether 0:14:4f:2a:9b:83 
                    bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
                            inet 63.192.77.16 netmask ffffff00 broadcast 63.192.77.255
                    
                    Nov  2 16:51:12 cstoc77022 bge: NOTICE: bge0: link up 100Mbps Full-Duplex
                    Nov  2 16:51:12 cstoc77022 in.mpathd[153]: The link has come up on bge0
                    Nov  2 16:51:12 cstoc77022 ip: WARNING: IP: Hardware address '00:14:4f:2a:9b:82' trying to be our a
                    ddress 063.192.077.021!
                    Nov  2 16:51:26 cstoc77022 in.mpathd[153]: NIC repair detected on bge0 of group test
                    Nov  2 16:51:26 cstoc77022 in.mpathd[153]: Successfully failed back to NIC bge0
                    Nov  2 16:51:34 cstoc77022 ip: WARNING: IP: Hardware address '00:14:4f:2a:9b:82' trying to be our a
                    ddress 063.192.077.022!
                    • 7. Re: IPMP failures on bge Interface
                      807559
                      1. Test your default router:
                      ping 63.192.77.9
                      2. Test another Sun boxes 63.192.77.1 and 63.192.77.236 and 63.192.77.19:
                      ping 63.192.77.1 ; ping 63.192.77.236 ; ping 63.192.77.19
                      3. If it works, add static routes and in a boot script:
                      route add -host 63.192.77.1 63.192.77.1 -static
                      route add -host 63.192.77.236 63.192.77.236 -static
                      route add -host 63.192.77.19 63.192.77.19 -static
                      4. Try your tests again.
                      5. If it does not work, install Recommended patches and bge patch (122027-08).
                      By the way, did your software uses arp for publishing MAC-IP addresses?
                      • 8. Re: IPMP failures on bge Interface
                        807559
                        I don't see how the explicit routes will change the results, but it's worth a try. Our software doesn't do anything with ARPs. The only thing we do is reduce the MTU size to make room for all the ESP headers. We've only had problems with the 'bge' interface, which is the mystery to us. Thanks for your help so far!
                        • 9. Re: IPMP failures on bge Interface
                          807559
                          The static routes didn't help so I installed the bge patch. It made my host unbootable, and since I'm using a Try&Buy T1000 there's no optical drive or external SCSI port. leaving a net install as my only option. I'm currently creating a JumpStart server, so hopefully my host will be back up for more testing later today.
                          • 10. Re: IPMP failures on bge Interface
                            807559
                            My host is back online, with a newer version of Solaris 10. It already has versions of the BGE patch, so I reran the IPMP tests. It now works normally for me. We'll test the corresponding patch for Solaris 8 also.

                            Thanks!