Click here to request your free 14-day trial of Cisco Umbrella through NetCraftsmen today!

10/23
2018
Carole Warner Reece

Troubleshooting ARP Issues with Routers and ASAs

My colleague and I have been supporting a customer who had complained about a new AdTran VoIP router added to his environment that was causing his ISP circuits to go down.

Their statement of the problem:

“We have a problem with our VoIP vendor and a router they are trying to add.  When added to our edge switch, it brings down both ISP circuits and so our network connections drop.  Can we engage you to assist with this issue so that we can get the vendor VoIP router connected?  This needs to be fairly immediate engagement as we are looking to deploy VoIP in less than a month and have been trying for almost a month to deploy their router.  We can put it in front of our firewall, not preferred, behind, etc.  They want to put it beside ours but that is the scenario that isn’t working.”

The customer provided a sketch of the environment that looked something like this:

After reviewing the switch and ASA configs, we updated the diagram to illustrate what we believed was actually configured, and shared it with the customer.

Note: We believed the customer had mis-labeled the VLANs going to his two ISP routers. We mentioned this to him.

VLAN Details

2960X-ISP#sh run
. . .
interface GigabitEthernet1/0/1
 description ASA 5545x-01 Port 0
 switchport access vlan 801
 switchport mode access
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 spanning-tree portfast
!
interface GigabitEthernet1/0/2
 description ASA 5545x-01 Port 1
 switchport access vlan 800
 switchport mode access
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 spanning-tree portfast
!
interface GigabitEthernet1/0/3
 description ASA 5545x-02 Port 0
 switchport access vlan 801
 switchport mode access
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 spanning-tree portfast
!
interface GigabitEthernet1/0/4
 description ASA 5545x-02 Port 1
 switchport access vlan 800
 switchport mode access
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 spanning-tree portfast
!
interface GigabitEthernet1/0/5
 description Port #2 on Adtran Router
 switchport access vlan 801
 switchport mode access
 shutdown
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 spanning-tree portfast
!
interface GigabitEthernet1/0/6
 description Port #3 on Adtran Router
 switchport access vlan 800
 switchport mode access
 shutdown
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 spanning-tree portfast
!
2960X-ISP#
2960X-ISP#sh mac address-table
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
. . .
 800    00c8.8b7c.970b    DYNAMIC     Gi1/0/2
 800    00c8.8b7c.9859    DYNAMIC     Gi1/0/4
 800    b2a8.6e7c.311f    DYNAMIC     Gi1/0/23  ! 192.168.18.1 on WAN-2
 801    0078.884b.a004    DYNAMIC     Gi1/0/22
 801    00c8.8b7c.970f    DYNAMIC     Gi1/0/1
 801    00c8.8b7c.985d    DYNAMIC     Gi1/0/3
 801    b0a8.6e7c.37d1    DYNAMIC     Gi1/0/24  ! 172.31.4.1 on WAN
Total Mac Addresses for this criterion: 27
2960X-ISP#

Note: The 2960X-ISP is a Layer 2 switch with no SVIs in VLAN 800 or 801.

The customer stated after connecting the AdTran router, the connections to the ISPs were dropped. He then needed to remove the AdTran router connections from the ISP switch, and clear the ARP table on the ASA. We never got a chance to see the issues while happening — the customer tested it without asking us to watch.

Show ARP from ASA5545 After Crash

After the crash, the customer reported that ASA5545 showed this show arp output:

ASA5545/act# sh arp
. . .
WAN 172.31.4.53 b0a8.6e7c.37c1 184
WAN 172.31.4.1  b0a8.6e7c.37c1 189
WAN 172.31.4.51 0078.884b.a004 9446
WAN 172.31.4.52 b0a8.6e7c.37c1 12397
WAN-2 192.168.18.1  b2a8.6e7c.311f 63
WAN-2 192.168.18.29 00c8.8b7c.970b 6067
WAN-2 192.168.18.8  b2a8.6e7c.311f 12367
WAN-2 192.168.18.26 b2a8.6e7c.311f 12367
WAN-2 192.168.18.23 b2a8.6e7c.311f 12367
. . .
ASA5545X/act#

Issue Summary

We were not sure why both ISP router MACs were showing up multiple times in the ASA ARP table. The ASA has the default proxy ARP defined on all interfaces. The new public IPs for the AdTran router were not configured in a NAT statement or network object group.

We heard that the customer initially provided the AdTran consultant with an incorrect IP address of 172.31.4.54 that was the IP of the secondary ASA in the firewall pair, but this was updated (The customer had not recorded the use of the secondary address in his IP spreadsheet).

The customer also had unused crypto map statements on four sets of ASAs pointing to 172.31.4.53 — the fifth ASA previously with address 172.31.4.53 had been removed from service.

We were wondering about the interaction of proxy ARP on the two ISP routers (both Juniper devices), the AdTran router, and the ASA5545 firewall, but we had not ever seen actual evidence of the issues (The log files on the ASA were too short to provide any details).

We asked the customer to remove the unused crypto map statements that pointed to 172.31.4.53.

We also asked for and received some AdTran show commands:

  • show interface so we could gather the MAC address of the new interfaces in case we needed to configure static ARP entries on the ASA
  • show run so we could see what else was configured on the AdTran

Note: The AdTran commands are very similar Cisco IOS. Neither the customer nor NetCraftsmen has access credential to the AdTran router.

After scanning the AdTran running-configuration, we noticed that there was a probe configured that used a non-local source address. It was using 172.31.4.54, which is IP address of the secondary ASA in the ASA pair.  We thought this was an error (We assumed the other failover stuff should work. The AdTran consultant did say that by design the secondary interface was not Layer 3 reachable when the primary interface was active).

Show Run from AdTran Router

AdTranVoIP#show run
. . .
!
ip local policy route-map Failover
!
probe Failover icmp-echo
destination 8.8.8.8
source-address 172.31.4.54
period 10
tolerance consecutive fail 5 pass 2
no shutdown
!
track Failover
test if probe Failover
no shutdown
!
interface gigabit-eth 0/1
description LAN
ip address  10.1.250.10  255.255.255.0
ip access-policy Private
media-gateway ip primary
no shutdown
!
!
interface gigabit-eth 0/2
description Verizon Connection
ip address  172.31.4.53  255.255.255.0
no ip proxy-arp
ip access-policy Public
media-gateway ip primary
qos-policy out VOIP
no shutdown
!
!
interface gigabit-eth 0/3
description Verizon Connection
ip address  192.168.18.28  255.255.255.0
no ip proxy-arp
ip access-policy "Secondary WAN"
media-gateway ip primary
qos-policy out VOIP
no shutdown
!
route-map Failover permit 1
match ip address Failover-ACL
set ip next-hop 172.31.4.1
set interface null 0
!
ip route 0.0.0.0 0.0.0.0 172.31.4.1 track Failover
ip route 0.0.0.0 0.0.0.0 192.168.18.1 50
!
. . .
AdTranVoIP#

Next Steps

The customer had the AdTran consultant update the probe source to use the physical address of the AdTran router.

During the next maintenance window, the customer reconnected the AdTran, but the AdTran consultant was not able to ping the primary interface of the AdTran router. Based on the show mac address-table information on the 2960X-ISP and our revised diagram, we convinced the customer to update the VLAN placement of the AdTran interfaces on the 2960X-ISP.

Afterwards, the AdTran consultant could ping the physical address of primary interface 172.31.4.53.

After waiting for 30 minutes or so for lurking problems but finding no issue, we surmised that after resolving the AdTran probe source IP and placing the AdTran interfaces in the correct VLANs on the 2960X-ISP switch, the AdTran router could be successfully added to the network without bringing down the ISP circuits. Success!

Our best guess is that previously the incorrect probe address of 172.31.4.54 was being proxied by the ISP router, and / or that the ASA objected to seeing echo reply packets sent to his secondary IP address without ever sending an echo request.

Summary:

Check the diagrams and assumptions of the customer. When possible, review the configuration of any new devices that cause the network failures. Also, do not allow probes to use IP addresses not assigned to the device especially if they overlap elsewhere in the network.

Carole Warner Reece

Architect

A senior network consultant with more than fifteen years of industry experience, Carole is one of our most highly experienced network professionals. Her current focus is on the data center and on network infrastructure.

View more Posts

 

Nick Kelly

Cybersecurity Engineer, Cisco

Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.

 

Virgilio “BONG” dela Cruz Jr.

CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data

Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.

 

John Cavanaugh

CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen

John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services.  Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.

He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.