3 Replies Latest reply: Jul 31, 2013 2:25 AM by user8007548 RSS

    Solaris 11.1 release, vlan mtu conectivity problem

    Crakeen
      Hi guys Im having conectivity problem with my setup. Im trying to configure my system like this:

      1 X86 server with 4 nics 2 bnx and 2 igb conected to HP v1910 switch configured as aggregattion with LACP Active and L2,L3 policy

      then I put some vlans over the aggregation VID 101 102 103

      and then i put some ip interfaces over each vlan and 1 ip interface on the aggregattion itself.

      The thing is , with all configured, everything goes fine, but when i change the mtu of the aggregatiion to something bigger than 1500, i loose conectivity on the vlan interfaces not in the aggrgatted ip.

      the courious thing is that when i start capturing packets on the vlan or in the aggregation with snoop (in promiscuous mode or not) everything start to work,, and as soon i stop the capture, all stop working again.

      My system info:

      root@BKP-SERVER:~# cat /etc/release

      Oracle Solaris 11.1 X86
      Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
      Assembled 19 September 2012

      root@BKP-SERVER:~# dladm show-phys
      LINK MEDIA STATE SPEED DUPLEX DEVICE
      net0 Ethernet up 1000 full bnx0
      net1 Ethernet up 1000 full bnx1
      net2 Ethernet up 1000 full igb0
      net3 Ethernet up 1000 full igb1

      root@BKP-SERVER:~# dladm show-link
      LINK CLASS MTU STATE OVER
      net0 phys 1500 up --
      net1 phys 1500 up --
      net2 phys 1500 up --
      net3 phys 1500 up --
      aggr1 aggr 1500 up net0 net1 net2 net3
      vlan101 vlan 1500 up aggr1
      vlan102 vlan 1500 up aggr1
      vlan103 vlan 1500 up aggr1

      root@BKP-SERVER:~# dladm show-vlan
      LINK VID OVER FLAGS
      vlan101 101 aggr1 -----
      vlan102 102 aggr1 -----
      vlan103 103 aggr1 -----


      root@BKP-SERVER:~# ipadm show-if
      IFNAME CLASS STATE ACTIVE OVER
      lo0 loopback ok yes --
      aggr1 ip ok yes --
      vlan101 ip ok yes --
      vlan102 ip ok yes --
      vlan103 ip ok yes --


      root@BKP-SERVER:~# ipadm show-addr
      ADDROBJ TYPE STATE ADDR
      lo0/v4 static ok 127.0.0.1/8
      aggr1/v4 static ok 10.9.0.43/16
      vlan101/v4 static ok 192.167.1.1/24
      vlan101/v4a static ok 192.167.1.2/24
      vlan101/v4b static ok 192.167.1.3/24
      vlan101/v4c static ok 192.167.1.4/24
      vlan102/v4 static ok 192.167.2.1/24
      vlan102/v4a static ok 192.167.2.2/24
      vlan102/v4b static ok 192.167.2.3/24
      vlan102/v4c static ok 192.167.2.4/24
      vlan103/v4 static ok 192.167.3.1/24
      vlan103/v4a static ok 192.167.3.2/24
      vlan103/v4b static ok 192.167.3.3/24
      vlan103/v4c static ok 192.167.3.4/24
      lo0/v6 static ok ::1/128

      pinging from mi S11 box to a server with ip 192.167.2.20 with already tested jumbo frames support and assigned to vlan 102

      root@BKP-SERVER:~# ping -s 192.167.2.20 1472 4
      PING 192.167.2.20: 1472 data bytes
      1480 bytes from 192.167.2.20: icmp_seq=0. time=0.485 ms
      1480 bytes from 192.167.2.20: icmp_seq=1. time=0.353 ms
      1480 bytes from 192.167.2.20: icmp_seq=2. time=0.528 ms
      1480 bytes from 192.167.2.20: icmp_seq=3. time=0.359 ms

      pinging from server over the aggregated link ip 10.9.0.43 to 10.9.1.1

      root@BKP-SERVER:~# ping -s 10.9.1.1 1472 4
      PING 10.9.1.1: 1472 data bytes
      1480 bytes from 10.9.1.1: icmp_seq=0. time=1.462 ms
      1480 bytes from 10.9.1.1: icmp_seq=1. time=2.959 ms
      1480 bytes from 10.9.1.1: icmp_seq=2. time=0.394 ms
      1480 bytes from 10.9.1.1: icmp_seq=3. time=0.330 ms

      ----10.9.1.1 PING Statistics----
      4 packets transmitted, 4 packets received, 0% packet loss
      round-trip (ms) min/avg/max/stddev = 0.330/1.286/2.959/1.230

      now i change the mtu ofthe aggregation:

      root@BKP-SERVER:~# dladm set-linkprop -p mtu=9000 aggr1

      root@BKP-SERVER:~# dladm show-link
      LINK CLASS MTU STATE OVER
      net0 phys 9000 up --
      net1 phys 9000 up --
      net2 phys 9000 up --
      net3 phys 9000 up --
      aggr1 aggr 9000 up net0 net1 net2 net3
      vlan101 vlan 1500 up aggr1
      vlan102 vlan 1500 up aggr1
      vlan103 vlan 1500 up aggr1

      now the ping over vlan:

      root@BKP-SERVER:~# ping -s 192.167.2.20 1472 4
      PING 192.167.2.20: 1472 data bytes

      ----192.167.2.20 PING Statistics----
      4 packets transmitted, 0 packets received, 100% packet loss

      now the ping over the aggregated interface

      root@BKP-SERVER:~# ping -s 10.9.1.1 1472 4
      PING 10.9.1.1: 1472 data bytes
      1480 bytes from 10.9.1.1: icmp_seq=0. time=0.496 ms
      1480 bytes from 10.9.1.1: icmp_seq=1. time=0.818 ms
      1480 bytes from 10.9.1.1: icmp_seq=2. time=0.422 ms
      1480 bytes from 10.9.1.1: icmp_seq=3. time=0.427 ms

      ----10.9.1.1 PING Statistics----
      4 packets transmitted, 4 packets received, 0% packet loss
      round-trip (ms) min/avg/max/stddev = 0.422/0.541/0.818/0.188

      now trying jumbo frames over vlan:

      root@BKP-SERVER:~# ping -s 192.167.2.20 8500 4
      PING 192.167.2.20: 8500 data bytes

      ----192.167.2.20 PING Statistics----
      4 packets transmitted, 0 packets received, 100% packet loss

      now trying jumbo frames over the aggregated link

      root@BKP-SERVER:~# ping -s 10.9.1.1 8500 4
      PING 10.9.1.1: 8500 data bytes
      8508 bytes from 10.9.1.1: icmp_seq=0. time=1.270 ms
      8508 bytes from 10.9.1.1: icmp_seq=1. time=0.922 ms
      8508 bytes from 10.9.1.1: icmp_seq=2. time=1.612 ms
      8508 bytes from 10.9.1.1: icmp_seq=3. time=0.859 ms

      ----10.9.1.1 PING Statistics----
      4 packets transmitted, 4 packets received, 0% packet loss
      round-trip (ms) min/avg/max/stddev = 0.859/1.166/1.612/0.348

      now as soon i start capture packets with any of this

      snoop -V -P -d aggr1 icmp (not promiscuous)
      snoop -V -P -d vlan102 icmp (not promiscuous)
      snoop -V -d aggr1 icmp
      snoop -V -d vlan102 icmp

      root@BKP-SERVER:~# ping -s 192.167.2.20 8500 4
      PING 192.167.2.20: 8500 data bytes
      8508 bytes from 192.167.2.20: icmp_seq=0. time=0.825 ms
      8508 bytes from 192.167.2.20: icmp_seq=1. time=0.781 ms
      8508 bytes from 192.167.2.20: icmp_seq=2. time=0.838 ms
      8508 bytes from 192.167.2.20: icmp_seq=3. time=0.787 ms

      ----192.167.2.20 PING Statistics----
      4 packets transmitted, 4 packets received, 0% packet loss
      round-trip (ms) min/avg/max/stddev = 0.781/0.808/0.838/0.028

      now i change the mtu on the vlans itselfs

      root@BKP-SERVER:~# dladm set-linkprop -p mtu=9000 vlan101
      root@BKP-SERVER:~# dladm set-linkprop -p mtu=9000 vlan102
      root@BKP-SERVER:~# dladm set-linkprop -p mtu=9000 vlan103

      root@BKP-SERVER:~# dladm show-link
      LINK CLASS MTU STATE OVER
      net0 phys 9000 up --
      net1 phys 9000 up --
      net2 phys 9000 up --
      net3 phys 9000 up --
      aggr1 aggr 9000 up net0 net1 net2 net3
      vlan101 vlan 9000 up aggr1
      vlan102 vlan 9000 up aggr1
      vlan103 vlan 9000 up aggr1

      ping over vlan again and same result

      root@BKP-SERVER:~# ping -s 192.167.2.20 8000 4
      PING 192.167.2.20: 8000 data bytes

      ----192.167.2.20 PING Statistics----
      4 packets transmitted, 0 packets received, 100% packet loss


      with packet capture on:

      root@BKP-SERVER:~# ping -s 192.167.2.20 8000 4
      PING 192.167.2.20: 8000 data bytes
      8008 bytes from 192.167.2.20: icmp_seq=0. time=0.928 ms
      8008 bytes from 192.167.2.20: icmp_seq=1. time=0.917 ms
      8008 bytes from 192.167.2.20: icmp_seq=2. time=0.930 ms
      8008 bytes from 192.167.2.20: icmp_seq=3. time=0.780 ms

      ----192.167.2.20 PING Statistics----
      4 packets transmitted, 4 packets received, 0% packet loss
      round-trip (ms) min/avg/max/stddev = 0.780/0.889/0.930/0.073

      packet capture output:

      192.167.2.1 -> 192.167.2.20 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.1 -> 192.167.2.20 IP D=192.167.2.20 S=192.167.2.1 LEN=8028, ID=52183, TOS=0x0, TTL=255
      192.167.2.1 -> 192.167.2.20 ICMP Echo request (ID: 1372 Sequence number: 3)
      ________________________________
      192.167.2.20 -> 192.167.2.1 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.20 -> 192.167.2.1 IP D=192.167.2.1 S=192.167.2.20 LEN=8028, ID=8028, TOS=0x0, TTL=255
      192.167.2.20 -> 192.167.2.1 ICMP Echo reply (ID: 1372 Sequence number: 3)
      ________________________________
      192.167.2.1 -> 192.167.2.20 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.1 -> 192.167.2.20 IP D=192.167.2.20 S=192.167.2.1 LEN=8028, ID=52187, TOS=0x0, TTL=255
      192.167.2.1 -> 192.167.2.20 ICMP Echo request (ID: 1373 Sequence number: 0)
      ________________________________
      192.167.2.20 -> 192.167.2.1 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.20 -> 192.167.2.1 IP D=192.167.2.1 S=192.167.2.20 LEN=8028, ID=8029, TOS=0x0, TTL=255
      192.167.2.20 -> 192.167.2.1 ICMP Echo reply (ID: 1373 Sequence number: 0)
      ________________________________
      192.167.2.1 -> 192.167.2.20 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.1 -> 192.167.2.20 IP D=192.167.2.20 S=192.167.2.1 LEN=8028, ID=52189, TOS=0x0, TTL=255
      192.167.2.1 -> 192.167.2.20 ICMP Echo request (ID: 1373 Sequence number: 1)
      ________________________________
      192.167.2.20 -> 192.167.2.1 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.20 -> 192.167.2.1 IP D=192.167.2.1 S=192.167.2.20 LEN=8028, ID=8030, TOS=0x0, TTL=255
      192.167.2.20 -> 192.167.2.1 ICMP Echo reply (ID: 1373 Sequence number: 1)
      ________________________________
      192.167.2.1 -> 192.167.2.20 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.1 -> 192.167.2.20 IP D=192.167.2.20 S=192.167.2.1 LEN=8028, ID=52190, TOS=0x0, TTL=255
      192.167.2.1 -> 192.167.2.20 ICMP Echo request (ID: 1373 Sequence number: 2)
      ________________________________
      192.167.2.20 -> 192.167.2.1 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.20 -> 192.167.2.1 IP D=192.167.2.1 S=192.167.2.20 LEN=8028, ID=8031, TOS=0x0, TTL=255
      192.167.2.20 -> 192.167.2.1 ICMP Echo reply (ID: 1373 Sequence number: 2)
      ________________________________
      192.167.2.1 -> 192.167.2.20 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.1 -> 192.167.2.20 IP D=192.167.2.20 S=192.167.2.1 LEN=8028, ID=52191, TOS=0x0, TTL=255
      192.167.2.1 -> 192.167.2.20 ICMP Echo request (ID: 1373 Sequence number: 3)
      ________________________________
      192.167.2.20 -> 192.167.2.1 ETHER Type=0800 (IP), VLAN ID=102, size=8046 bytes
      192.167.2.20 -> 192.167.2.1 IP D=192.167.2.1 S=192.167.2.20 LEN=8028, ID=8032, TOS=0x0, TTL=255
      192.167.2.20 -> 192.167.2.1 ICMP Echo reply (ID: 1373 Sequence number: 3)

      there is any solution for this problem? or there is any related bug for this?

      any idea? :(

      Ty very much in advance and sorry for my english.

      Edited by: 957695 on 10-may-2013 6:00