New Nexus 9K Items
I was recently troubleshooting a vPC port channel issue with a pair of Nexus 5500 switches that were trying to implement new port channels to two NetApp controllers.
I got involved when my storage friend sent me an email one evening asking me to check the port channel for one interface that was not wanting to participate in the port channel from the NetApp perspective. As it turned out, I was not initially able to contact him for more details because we were working different shifts.
I checked on the N5Ks, and I saw that one port channel was down on ST-5K1, but all of the other port channels were up.
The vPC results showed that the ST-5Ks thought two different devices were connected to their respective e1/32 interfaces:
I checked the recent log messages for the interface. Maybe someone had pulled out the SFP?
Hmm — both 5Ks had complained about a hot standby state at the same time. I usually find this means the cables have been cross connected so the vPC port channel from the Nexus switch pair is connected to two different devices, which is not a supported state.
I was also wondering why the ST-5K1 17:25:45 log entry said the hardware type changed to No-Transceiver. This seemed off — I thought all the cables had been placed earlier. My guess was that no one had removed the SFP, and that maybe NX-OS was confused about the current state of the cables.
To see if I could clear out the ‘vpc port channel mis-config,’ I bounced the port channel interface.
I checked, and the vPC was still listed as misconfigured.
In addition, the physical interface had a new weird state:
I do not recall ever seeing a physical interface down due to adminCfgChange as the reason. But I interpreted the state to mean the NX-OS was confused about the interface state based on the configuration changes, so I also shut/no shut the physical interface e1/32.
There were a lot of messages over the next 30 seconds or so. After all the initialization, both the interface and the port channel came up.
I sent a note to the storage team and they did confirm the port channels were now appearing correctly.
My guess was the NX-OS on ST-5K1 was confused about the vPC port channel configuration due to cables that were incorrectly connected to two devices, then hot-swapped to the correct orientation. ST-5K1 appears to have needed the physical interface to be reset for the incorrect vPC port channel configuration to get cleared out. Possibly based on the timing or order of the cable swaps, ST-5K2 recovered on its own.
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.