Click here to request your free 14-day trial of Cisco Umbrella through NetCraftsmen today!

One of my customers recently had an STP issue in his remote data center. He was not sure of all the details, but sometime during or after the migration of some Layer 2 links between his N7K and N5Ks into port-channels (possibly while VMware infrstructure changes were being made) his network had problems. The network team disconnected part of the port-channel configuration, and the network stablized. However, his remaining network status troubled him.

Planned State
The planned network design was to look like the following diagram:

2012_08_13_nexus port-channels-planned

This seems like a reasonable design. I’m not sure if it was completely implemented, or what specific network issue he had. I only had a chance to look at the network after the fact.

Current State
When I looked at the network, the following topology was in place.

2012_08_13_nexus port-channels-current

The network team had removed channel-group 1001 from Eth1/5 on 5K2 and from Eth 1/1 on N7K1. The show port-channel summary command provides a quick view of the topology:

5K1# sh port-c sum
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
10    Po10(SU)    Eth      NONE      Eth1/9(P)   Eth1/10(P)   
. . .
1001  Po1001(SU)  Eth      NONE      Eth1/1(P)   
5K1#

. . .

5K2# sh port-c s
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
10    Po10(SU)    Eth      NONE      Eth1/9(P)   Eth1/10(P)   
. . .
1001  Po1001(SD)  Eth      NONE      --
5K2#

. . .

7K1# sh port-c s
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
1001  Po1001(SU)  Eth      NONE      Eth1/5(P)    
7K1#

Does this seem reasonable to you? Are you maybe concerned that  a port-channel is being formed from 7K1 through 5K2 to N5K1?

Explanation
Looking at the previous show port-channel summary commands and the interface configurations helps illustrate that this not alarming, it is default NX-OS behavior.

5K1# sh run int e1/1

!Command: show running-config interface Ethernet1/1
!Time: Fri Aug 10 15:22:10 2012

version 5.1(3)N1(1)

interface Ethernet1/1
switchport mode trunk
channel-group 1001

5K1#

. . .

5K2# sh run int e1/5

!Command: show running-config interface Ethernet1/5
!Time: Fri Aug 10 15:23:17 2012

version 5.1(3)N1(1)

interface Ethernet1/5
switchport mode trunk

5K2#

. . .

7K1# sh run int e1/1,e1/5

!Command: show running-config interface Ethernet1/1, Ethernet1/5
!Time: Fri Aug 10 15:24:27 2012

version 5.1(3)N1(1)
interface Ethernet1/1
switchport
switchport mode trunk
no shutdown

interface Ethernet1/5
switchport
switchport mode trunk
channel-group 1001
no shutdown

7K1#

The “channel-group 1001” NX-OS interface command defaults to mode “on” which shows as “NONE” as protocol in the show port-channel summary command output. Both 5K1 and 7K1 are currently configured with channel-group mode “ON” and are NOT using LACP for their port-channels.

Both the passive and active LACP modes allow LACP to negotiate between LAN ports to determine if they can form an EtherChannel, based on criteria such as port speed and trunking state. The ON state does not care, it brings up the port-channel even if the remote end is simply a Layer 2 connection.

If instead you use “channel-group 1001 mode active” NX-OS interface command when LACP is enabled, then both sides of the link negotiate, and show LACP as the protocol, and I, P, or D as the member state depending if the other side is similarly configured.

I  re-configured all port-channels as “mode active”, brought up all the port-channels, and then removed port-channel 1001 from 5K2 Ethe 1/5 and 7K1 Eteh 1/1. Now you can see a more illustrative result from the show port-channel summary commands:

5K1# sh port-c s
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
10    Po10(SU)    Eth      LACP      Eth1/9(P)    Eth1/10(P)
1001  Po1001(SD)  Eth      LACP      Eth1/1(I)
5K1(config-if)#

. . .

5K2# sh port-c s
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
10    Po10(SU)    Eth      LACP      Eth1/9(P)    Eth1/10(P)
1001  Po1001(SD)  Eth      NONE      --
5K2#

. . .

7K1# sh port-c s
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
Channel
-------------------------------------------------------------------------------
1001  Po1001(SD)  Eth      LACP      Eth1/5(s)
7K1#


Recommendations
I recommended my customer follows a Cisco best practice and use LACP with his port-channel deployment. Port-channel to vPC configurations can be complex, and LACP helps support two goals:

    • Allows the topology converge into a usable topology (even in the presence of a configuration error)
    • Provides more descriptive information that helps locate potential problems

— cwr

___________________________________________________________________

If you would like some additional details on vPCs, the following references may be helpful:

Carole Warner Reece

Architect

A senior network consultant with more than fifteen years of industry experience, Carole is one of our most highly experienced network professionals. Her current focus is on the data center and on network infrastructure.

View more Posts

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.