New Nexus 9K Items
I was recently troubleshooting an SVI issue with a pair Nexus 7000 switches. The switches were a vPC pair, and had been set up a couple of years ago. I got involved when my customer mentioned that he did not know why an SVI was down on one of the N7Ks.
The “down” SVI was on a non-vPC VLAN that was carried on a trunk parallel to the peer-link.
I initially thought that the VLAN for the SVI may have been deleted, so asked the customer to check if the VLAN was configured. He verified the VLAN was configured with assigned ports.
We checked the physical interface status for the port-channel — the interface for the down SVI was connected.
We looked at the spanning tree status for the VLAN on both switches:
OK, the SVI was down because the STP state was blocking. Next question — why was the STP role backup and the STP state blocking?
A quick review of STP definitions may help:
So why was Po2 a backup port for STP? It must be receiving more useful BPDUs from the same bridge it is on.
We knew that ST1 and ST2 are vPC peers.
The vPC Best Practices Design Guide states:
When vPC peer-switch is activated, it is mandatory that both peer devices have the exact same spanning tree configuration and more precisely the same Spanning Tree Protocol priority for all vPC VLANs.
From the Cisco Nexus 7000 Peer-Switch Configuration (Hybrid Setup) document, we saw:
With peer-switch enabled, each Nexus 7000 switch generates BPDUs with the root bridge set to the virtual bridge ID and the designated bridge set to the physical bridge ID. Since the priorities are the same, all non-vPC connections always forward on the link connected to the Nexus 7000 switch with the lower bridge ID (N7K-1 in this example) and block on the links connected to the Nexus 7000 switch with the higher bridge ID (N7K-2 in this example).
We checked, and the vPC configuration included the peer-switch command.
Another vPC practice is that the non-vPC VLANs use a higher STP priority on the secondary switch. However, from the earlier show spanning-tree vlan 999 results, we saw both switches are using priority 8192.
Note: Cisco also recommends using spanning-tree pseduo-information configurations to enable load balancing between the VLANs on non-VPC links to other switches. This additional configurations does not seem to be needed for this VLAN as long as a different spanning-tree vlan priority is used for the non-vPC VLANs on the secondary switch.
Since the SVI had been working in the past, the spanning-tree VLAN priority for VLAN 999 had been changed, possibly when a new VLAN was added to the vPC pair.
We updated the STP VLAN priority on ST2:
This was indeed the issue. With this update configured, the STP role changed to Root with status forwarding, and the SVI came up.
I believe the issue occurred when a new VLAN was added to the network, and the exact spanning-tree vlan priority commands on 7K1 were mistakenly copied to 7K2.
You need to be careful when you have vPC VLANs and non-vPC VLANs, and know how to configure each kind on the primary and secondary vPC switches.
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.