network link shows down and comes up its own after 25-30secs. We are using IPMP and cluster on T2-7 — oracle-tech

    Forum Stats

  • 3,708,780 Users
  • 2,241,126 Discussions
  • 7,840,613 Comments

Discussions

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

network link shows down and comes up its own after 25-30secs. We are using IPMP and cluster on T2-7

/var/adm/messages:Dec 19 16:43:23 VMHOST07 mac: [ID 486395 kern.info] NOTICE: ixgbe0 link down

/var/adm/messages:Dec 19 16:43:23 VMHOST07 in.mpathd[68]: [ID 215189 daemon.error] The link has gone down on pcie4_p0

/var/adm/messages:Dec 19 16:43:31 VMHOST07 mac: [ID 435574 kern.info] NOTICE: ixgbe0 link up, 1000 Mbps, full duplex

/var/adm/messages:Dec 19 16:43:44 VMHOST07 in.mpathd[68]: [ID 820239 daemon.error] The link has come up on pcie4_p0

/var/adm/messages.1:Dec  8 08:56:15 VMHOST07 mac: [ID 486395 kern.info] NOTICE: ixgbe4 link down

/var/adm/messages.1:Dec  8 10:07:41 VMHOST07 mac: [ID 435574 kern.info] NOTICE: ixgbe4 link up, 10000 Mbps, full duplex

/var/adm/messages.1:Dec  9 00:01:46 VMHOST07 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 1000 Mbps, full duplex

/var/adm/messages.1:Dec  9 19:20:20 VMHOST07 mac: [ID 486395 kern.info] NOTICE: ixgbe0 link down

/var/adm/messages.1:Dec  9 19:20:20 VMHOST07 in.mpathd[68]: [ID 215189 daemon.error] The link has gone down on pcie4_p0

/var/adm/messages:Dec 20 02:47:00 VMHOST08 mac: [ID 486395 kern.info] NOTICE: ixgbe0 link down

/var/adm/messages:Dec 20 02:47:00 VMHOST08 in.mpathd[68]: [ID 215189 daemon.error] The link has gone down on pcie4_p0

/var/adm/messages:Dec 20 02:47:07 VMHOST08 mac: [ID 435574 kern.info] NOTICE: ixgbe0 link up, 1000 Mbps, full duplex

/var/adm/messages:Dec 20 02:47:21 VMHOST08 in.mpathd[68]: [ID 820239 daemon.error] The link has come up on pcie4_p0

We have noticed intermittent link down/up events with different times and on different cluster nodes. Wondering if this is something to do with IPMP ? as physical cables has been tested and looking good. We have links can go 10GBPS as auto-negotiation it set.

Answers

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    what causes this when talk to Cisco 6500

    mac: [ID 486395 kern.notice] NOTICE: ixgbe0 link down

    up down every 5 seconds

  • NikNik Posts: 2,700 Bronze Crown

    Hi.

    This problem specific for hardware, may be OS-levelproblem, but it's not Enterpsise Manager problem...


    Message " XXX link down" means that this port lost signal of "media-present".

    This can be caused:

    • Bad network port on server side
    • Bad network port on switch side
    • Bad cable between ports
    • Some ports misconfigugaration.


    You say that links is 10Gbps but it's up only on 1000 Mbps. Is it expected behavior ? What switch model was used?

    Check messages on switch side. May be switch for some reason restart this ports. How You test cables to switch ?

    What level of cable was used ?

    Check output of command eeprom local-mac-address? on Solaris. This command should return "True"


    Reagrds,

    Nik

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon


    Yup eeprom set to true

    LACP active and changed to long

    funny part on 01 ixgbe0 is table and ixgbe1 flips.

    dladm shows the ixgbe0 and ixgbe1 have same mac which seems to be the compliant from the CISCO 6500

    But Oracles instruction on the site to change the NAC doesn't work ( Normal for Oracle doc's )



    Jan 11 15:36:19 sun-bsu-02 mac: [ID 435574 kern.notice] NOTICE: ixgbe0 link up, 10000 Mbps, full duplex

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 486395 kern.notice] NOTICE: ixgbe0 link down

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 435574 kern.notice] NOTICE: ixgbe0 link up, 10000 Mbps, full duplex

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 486395 kern.notice] NOTICE: ixgbe0 link down

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 435574 kern.notice] NOTICE: ixgbe0 link up, 10000 Mbps, full duplex

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 486395 kern.notice] NOTICE: ixgbe0 link down

    Jan 11 15:36:20 sun-bsu-02 mac: [ID 435574 kern.notice] NOTICE: ixgbe0 link up, 10000 Mbps, full duplex

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    we have 3 servers 01 02 03

    01 has ixgbe1 flipping and box 2 has ixgbe0 flipping ( the servers have different MAC's )

    but the interface on each servers has the same MAC

  • NikNik Posts: 2,700 Bronze Crown

    Hi.

    With local-mac-addres?=true - every NIC should have unique MAC address. Try reboot server, becouse this variable have effect only at boot time.

    IPMP - it's active-passive NIC reservation.

    LACP -it's 802.3 link aggregation.

    It's different type of reservation so switch side and server side should use same type of settings.

    Show real used configuration from Solaris and swtich side.

    What Solaris version used?


    Regards,

    Nik

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    Solaris 11.4.25

    rebooted several times how many times do I need to reboot for it to take affect ?

    even power cycled t 3 times

  • NikNik Posts: 2,700 Bronze Crown
    edited January 11

    Hi.

    It's public forum where one admin try help to other. In case You have support contract - open SR for this issue.

    I can see only what you say.

    So in case You need resolve this issue - provide more information:


    ifconfig -a
    dladm show-link
    dladm show-phys
    eeprom local-mac-address?
    

    configuration of switch ports that connected to this server.


    As test - disconnect (shutdown on switch side) one ethernet port for one server.

    In case it's stop link issue on working port - it's mean that You have configuration problem.

    In case You still have link issue - it's mean that you have some hardware problem.


    You wrote:

    But Oracles instruction on the site to change the NAC doesn't work ( Normal for Oracle doc's )

    What steps You do and what Doc was used ?

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    [email protected]:~# ifconfig -a

    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index                                1

           inet 127.0.0.1 netmask ff000000

    sp-phys0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mt                               u 1500 index 2

           inet 169.254.182.77 netmask ffffff00 broadcast 169.254.182.255

           ether 2:21:28:57:47:17

    pop592002: flags=100201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,PHYSRUNNIN                               G> mtu 1500 index 4

           inet 10.56.92.26 netmask ffffff00 broadcast 10.56.92.255

           ether 68:5:ca:c3:8c:bc

    pop730002: flags=100201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,PHYSRUNNIN                               G> mtu 1500 index 3

           inet 10.0.73.26 netmask ffffff00 broadcast 10.0.73.255

           ether 68:5:ca:c3:8c:bc

    lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index                                1

           inet6 ::1/128

    sp-phys0: flags=120002000840<RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index                                2

           inet6 ::/0

           ether 2:21:28:57:47:17

    pop592002: flags=120202000840<RUNNING,MULTICAST,IPv6,CoS,PHYSRUNNING> mtu 1500 i                               ndex 4

           inet6 ::/0

           ether 68:5:ca:c3:8c:bc

    pop730002: flags=120202000840<RUNNING,MULTICAST,IPv6,CoS,PHYSRUNNING> mtu 1500 i                               ndex 3

           inet6 ::/0

           ether 68:5:ca:c3:8c:bc

    [email protected]:~# dladm show-link

    LINK               CLASS    MTU   STATE   OVER

    net0               phys     1500  down    --

    net1               phys     1500  down    --

    net2               phys     1500  down    --

    net3               phys     1500  down    --

    net4               phys     1500  up      --

    net5               phys     1500  up      --

    pop2               aggr     1500  up      net4 net5

    pop592002          vlan     1500  up      pop2

    pop730002          vlan     1500  up      pop2

    sp-phys0           phys     1500  up      --

    [email protected]:~# dladm show-phys

    LINK           MEDIA        STATE     SPEED DUPLEX   DEVICE

    net0           Ethernet     down      0     unknown  i40e0

    net1           Ethernet     down      0     unknown  i40e1

    net2           Ethernet     down      0     unknown  i40e2

    net3           Ethernet     down      0     unknown  i40e3

    net4           Ethernet     up        10000 full     ixgbe0

    net5           Ethernet     up        10000 full     ixgbe1

    sp-phys0       Ethernet     up        10    full     usbecm2

    [email protected]:~# eeprom local-mac-address?

    local-mac-address?=true

    [email protected]:~#

    net4    mac-address    rw  68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc --

    net5    mac-address    rw  68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bd --

    pop2    mac-address    rw  68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc --

    pop592002 mac-address   r-  68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc --

    pop730002 mac-address   r-  68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc 68:5:ca:c3:8c:bc --

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    the problem the switch doesn't like them all having the same mac.

    The dladm set-linkprop -p mac-address= net4 ( or pop59022002 ) according to E23824-1 doesn't work

  • NikNik Posts: 2,700 Bronze Crown
    edited January 11

    This configuration is not IPMP. It's link aggragation LACP (802.3).

    This configuration require correct configuration on switch side (Enable autoconfigure LACP channels or configure etherchannes) .

    You not have problem with duplicated MAC.

    pop592002  and pop730002  are VLANs on pop2. It's share same physical media so have same MAC.


    What about configuration from switch?    

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    Cisco guy says it is the SUN.

    That the switch is configured for Both VLANS and LACP

  • User_GMV2WUser_GMV2W Posts: 8 Red Ribbon

    The switch sees mac AA on port 1/1 then sees mac AA on port 9/ and drops the one on port 9/1

    second or 3 later it brings up the one on port 9//1 see s the same mac and drops it. over and over and over

  • NikNik Posts: 2,700 Bronze Crown
    edited January 12

    Hi.

    You describe a typical mac-flapping problem. This cause that cisco temporary shutdown one port.


    On Cisco side You should see etherchannel from ports 1/1 ; 9/1, but now it's works as 2 undepended ports.

    Configuration for both side of LACP aggregation should be consistent. ( Which side is active etc).


    For resolve this issue You should work with Cisco guys for analyze logs on Cisco side and etherchannel link status.

    As start point: https://community.cisco.com/t5/other-network-architecture/etherchannel-and-solaris-10/td-p/659192

    Also read: Solaris Link Aggregation First Checks (Doc ID 2169399.1)


    Show aggregate configuration on Solaris side ( dladm show-aggr )

    What configuration expected ? ( LACP (802.3) or IPMP (DLMP) ? )

Sign In or Register to comment.