Click here to request your free 14-day trial of Cisco Umbrella through NetCraftsmen today!

9/15
2016
Carole Warner Reece

Troubleshooting HSRP on Cisco Nexus 5500 Switches

I was recently troubleshooting an HSRP issue with two Nexus 5500 switches that both were in the active state. (I was trying to add HSRP to a long-running pair of Nexus 5500s.)


HQ-ST-5K1# sh hsrp brie
                     P indicates configured to preempt.
                     |
Interface   Grp Prio P State    Active addr      Standby addr     Group addr
Vlan230     43  110  P Active   local            unknown          172.20.23.1  (conf)
HQ-ST-5K1#
HQ-ST-5K1# sh run int vlan 230
 
. . .
 
interface Vlan230
  description NFS_172.20.23.0/25
  no shutdown
  ip address 172.20.23.2/25
  hsrp 43
    ip 172.20.23.1
   preempt
    priority 110
 
HQ-ST-5K1#

 

HQ-ST-5K2# sh hsrp brie
P indicates configured to preempt.
|
Interface   Grp Prio P State    Active addr      Standby addr     Group addr
Vlan230     43  100    Acti     local            unknown   172.20.23.1    (conf)
HQ-ST-5K2#
HQ-ST-5K2# sh run int vlan 230
 
. . .
 
interface Vlan230
description NFS_172.20.23.0/25
no shutdown
ip address 172.20.23.3/25
hsrp 43
ip 172.20.23.1
 
HQ-ST-5K2#

The two 5500 switches were a vPC pair, and I had connectivity on the VLAN between them. The vPC peering looked fine.

I looked at debugs on the HSRP engine packets, and noticed that while both devices were sending out hello packets, neither device was receiving the other device’s hello packets.

I saw results like these:

HQ-ST-5K1# deb hsrp engine pack hello

2016 Aug  2 18:30:19.637063 hsrp: Vlan230[43/V4]: Hello out Active pri 100 ip 172.20.23.1
2016 Aug  2 18:30:19.637443 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  2 18:30:22.637064 hsrp: Vlan230[43/V4]: Hello out Active pri 100 ip 172.20.23.1
2016 Aug  2 18:30:22.637446 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  2 18:30:25.637078 hsrp: Vlan230[43/V4]: Hello out Active pri 100 ip 172.20.23.1
2016 Aug  2 18:30:25.637819 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  2 18:30:28.636996 hsrp: Vlan230[43/V4]: Hello out Active pri 100 ip 172.20.23.1
2016 Aug  2 18:30:28.637376 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco

I also checked the status of the forwarding engine on the Nexus 5500s with the show module command.

HQ-ST-5K1# show module

Mod Ports Module-Type                         Model                  Status
--- ----- ----------------------------------- ---------------------- ------------
1   32    O2 32X10GE/Modular Universal Platfo N5K-C5548UP-SUP        active *
2   16    O2 16X10GE Ethernet Module          N55-M16P               ok
3   0     O2 Daughter Card with L3 ASIC       N55-D160L3-V2          offline

Mod  Sw              Hw      World-Wide-Name(s) (WWN)
---  --------------  ------  ----------------------------------------------------
1    6.0(2)N1(2)     1.0     --
2    6.0(2)N1(2)     1.0     --
3    6.0(2)N1(2)     1.0     --
 
. . .
 
HQ-ST-5K2#

Hmm – I saw that the Layer 3 forwarding card for both Nexus 5500s was offline.

I did some searching on the Internet. I saw a mention on the Cisco Licensing guidelines here:

‘When you configure HSRP in a virtual port channel (vPC) on the Nexus 6000 platform without installing the LAN_BASE_SERVICES_PKG license, it sends an HSRP hello message to the link local multicast address and enables the Active-Active state.’ 

This seemed like it was possibly the same issue with the Nexus 5500s. The Layer 3 forwarding card really should be online.

I checked the state of the licenses on both of the Nexus 5500s with the show license command.

HQ-ST-5K1# sh license

HQ-ST-5K1#
 
. . .
 
HQ-ST-5K2# sh license
HQ-ST-5K2#

Neither of the Nexus 5500s has an installed license. So assuming the root cause was a missing LAN_Base license, I worked through the process to obtain, load, and install the LAN_Base license. (For the process, please review my recent blog post on “Adding a License File to a Cisco Nexus 5500 Switch.”)

After installing the licenses, I found that HSRP worked properly and the Layer 3 forwarding engine was online.

HQ-ST-5K1# sh mod
Mod Ports Module-Type                         Model                  Status
--- ----- ----------------------------------- ---------------------- ------------
1   32    O2 32X10GE/Modular Universal Platfo N5K-C5548UP-SUP        active *
2   16    O2 16X10GE Ethernet Module          N55-M16P               ok
3   0     O2 Daughter Card with L3 ASIC       N55-D160L3-V2          ok
 
Mod  Sw              Hw      World-Wide-Name(s) (WWN)
---  --------------  ------  ----------------------------------------------------
1    6.0(2)N1(2)     1.0     --
2    6.0(2)N1(2)     1.0     --
3    6.0(2)N1(2)     1.0     --

. . .
HQ-ST-5K1#
HQ-ST-5K1# sh hsrp brie
P indicates configured to preempt.
|
Interface   Grp Prio P State    Active addr      Standby addr     Group addr
Vlan230    43  110  P Active   local            172.20.23.3     172.20.23.1    (conf)
HQ-ST-5K1#
 
HQ-ST-5K2# sh hsrp brie
P indicates configured to preempt.
|
Interface   Grp Prio P State    Active addr      Standby addr     Group addr
Vlan2430    43  100    Standby  172.20.243.2     local            172.20.243.1    (conf)
HQ-ST-5K2#
HQ-ST-5K2# sh mod
Mod Ports Module-Type                         Model                  Status
--- ----- ----------------------------------- ---------------------- ------------
1   32    O2 32X10GE/Modular Universal Platfo N5K-C5548UP-SUP        active *
2   16    O2 16X10GE Ethernet Module          N55-M16P               ok
3   0     O2 Daughter Card with L3 ASIC       N55-D160L3-V2          ok

. . .
HQ-ST-5K2

The debug results also showed the inbound hellos.


HQ-ST-5K1# deb hsrp engine pack hello
2016 Aug  4 17:49:40.340104 hsrp: Vlan230[43/V4]: Hello out Active pri 110 ip 172.20.23.1
2016 Aug  4 17:49:40.340469 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  4 17:49:40.342102 hsrp: Vlan230[43/V4]: Hello in from 172.20.23.3 State Standby pri 100 ip 172.20.23.1
2016 Aug  4 17:49:40.342475 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  4 17:49:43.341535 hsrp: Vlan230[43/V4]: Hello in from 172.20.23.3 State Standby pri 100 ip 172.20.23.1
2016 Aug  4 17:49:43.341896 hsrp: Vlan230[43/V4]: hel 3 hol 10 auth cisco
2016 Aug  5 17:49:43.343839 hsrp: Vlan230[43/V4]: Hello out Active pri 110 ip 172.20.23.1
2016 Aug  4 17:49:46.347831 hsrp: Vlan230[43/V4]: Hello in from 172.20.23.3 State Standby pri 100 ip 172.20.23.1
HQ-ST-5K1# un all

The Nexus 5500 pair had worked great for over a year until we needed HSRP. Then we needed to install the appropriate license.

Carole Warner Reece

Architect

A senior network consultant with more than fifteen years of industry experience, Carole is one of our most highly experienced network professionals. Her current focus is on the data center and on network infrastructure.

View more Posts

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.