My colleague and I have been supporting a customer who had complained about a new AdTran VoIP router added to his environment that was causing his ISP circuits to go down.
Their statement of the problem:
“We have a problem with our VoIP vendor and a router they are trying to add. When added to our edge switch, it brings down both ISP circuits and so our network connections drop. Can we engage you to assist with this issue so that we can get the vendor VoIP router connected? This needs to be fairly immediate engagement as we are looking to deploy VoIP in less than a month and have been trying for almost a month to deploy their router. We can put it in front of our firewall, not preferred, behind, etc. They want to put it beside ours but that is the scenario that isn’t working.”
The customer provided a sketch of the environment that looked something like this:
After reviewing the switch and ASA configs, we updated the diagram to illustrate what we believed was actually configured, and shared it with the customer.
Note: We believed the customer had mis-labeled the VLANs going to his two ISP routers. We mentioned this to him.
VLAN Details
2960X-ISP#sh run . . . interface GigabitEthernet1/0/1 description ASA 5545x-01 Port 0 switchport access vlan 801 switchport mode access srr-queue bandwidth share 1 30 35 5 priority-queue out mls qos trust dscp spanning-tree portfast ! interface GigabitEthernet1/0/2 description ASA 5545x-01 Port 1 switchport access vlan 800 switchport mode access srr-queue bandwidth share 1 30 35 5 priority-queue out mls qos trust dscp spanning-tree portfast ! interface GigabitEthernet1/0/3 description ASA 5545x-02 Port 0 switchport access vlan 801 switchport mode access srr-queue bandwidth share 1 30 35 5 priority-queue out mls qos trust dscp spanning-tree portfast ! interface GigabitEthernet1/0/4 description ASA 5545x-02 Port 1 switchport access vlan 800 switchport mode access srr-queue bandwidth share 1 30 35 5 priority-queue out mls qos trust dscp spanning-tree portfast ! interface GigabitEthernet1/0/5 description Port #2 on Adtran Router switchport access vlan 801 switchport mode access shutdown srr-queue bandwidth share 1 30 35 5 priority-queue out spanning-tree portfast ! interface GigabitEthernet1/0/6 description Port #3 on Adtran Router switchport access vlan 800 switchport mode access shutdown srr-queue bandwidth share 1 30 35 5 priority-queue out spanning-tree portfast ! 2960X-ISP# 2960X-ISP#sh mac address-table Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---- ----------- -------- ----- . . . 800 00c8.8b7c.970b DYNAMIC Gi1/0/2 800 00c8.8b7c.9859 DYNAMIC Gi1/0/4 800 b2a8.6e7c.311f DYNAMIC Gi1/0/23 ! 192.168.18.1 on WAN-2 801 0078.884b.a004 DYNAMIC Gi1/0/22 801 00c8.8b7c.970f DYNAMIC Gi1/0/1 801 00c8.8b7c.985d DYNAMIC Gi1/0/3 801 b0a8.6e7c.37d1 DYNAMIC Gi1/0/24 ! 172.31.4.1 on WAN Total Mac Addresses for this criterion: 27 2960X-ISP#
Note: The 2960X-ISP is a Layer 2 switch with no SVIs in VLAN 800 or 801.
The customer stated after connecting the AdTran router, the connections to the ISPs were dropped. He then needed to remove the AdTran router connections from the ISP switch, and clear the ARP table on the ASA. We never got a chance to see the issues while happening — the customer tested it without asking us to watch.
Show ARP from ASA5545 After Crash
After the crash, the customer reported that ASA5545 showed this show arp output:
ASA5545/act# sh arp . . . WAN 172.31.4.53 b0a8.6e7c.37c1 184 WAN 172.31.4.1 b0a8.6e7c.37c1 189 WAN 172.31.4.51 0078.884b.a004 9446 WAN 172.31.4.52 b0a8.6e7c.37c1 12397 WAN-2 192.168.18.1 b2a8.6e7c.311f 63 WAN-2 192.168.18.29 00c8.8b7c.970b 6067 WAN-2 192.168.18.8 b2a8.6e7c.311f 12367 WAN-2 192.168.18.26 b2a8.6e7c.311f 12367 WAN-2 192.168.18.23 b2a8.6e7c.311f 12367 . . . ASA5545X/act#
Issue Summary
We were not sure why both ISP router MACs were showing up multiple times in the ASA ARP table. The ASA has the default proxy ARP defined on all interfaces. The new public IPs for the AdTran router were not configured in a NAT statement or network object group.
We heard that the customer initially provided the AdTran consultant with an incorrect IP address of 172.31.4.54 that was the IP of the secondary ASA in the firewall pair, but this was updated (The customer had not recorded the use of the secondary address in his IP spreadsheet).
The customer also had unused crypto map statements on four sets of ASAs pointing to 172.31.4.53 — the fifth ASA previously with address 172.31.4.53 had been removed from service.
We were wondering about the interaction of proxy ARP on the two ISP routers (both Juniper devices), the AdTran router, and the ASA5545 firewall, but we had not ever seen actual evidence of the issues (The log files on the ASA were too short to provide any details).
We asked the customer to remove the unused crypto map statements that pointed to 172.31.4.53.
We also asked for and received some AdTran show commands:
- show interface so we could gather the MAC address of the new interfaces in case we needed to configure static ARP entries on the ASA
- show run so we could see what else was configured on the AdTran
Note: The AdTran commands are very similar Cisco IOS. Neither the customer nor NetCraftsmen has access credential to the AdTran router.
After scanning the AdTran running-configuration, we noticed that there was a probe configured that used a non-local source address. It was using 172.31.4.54, which is IP address of the secondary ASA in the ASA pair. We thought this was an error (We assumed the other failover stuff should work. The AdTran consultant did say that by design the secondary interface was not Layer 3 reachable when the primary interface was active).
Show Run from AdTran Router
AdTranVoIP#show run . . . ! ip local policy route-map Failover ! probe Failover icmp-echo destination 8.8.8.8 source-address 172.31.4.54 period 10 tolerance consecutive fail 5 pass 2 no shutdown ! track Failover test if probe Failover no shutdown ! interface gigabit-eth 0/1 description LAN ip address 10.1.250.10 255.255.255.0 ip access-policy Private media-gateway ip primary no shutdown ! ! interface gigabit-eth 0/2 description Verizon Connection ip address 172.31.4.53 255.255.255.0 no ip proxy-arp ip access-policy Public media-gateway ip primary qos-policy out VOIP no shutdown ! ! interface gigabit-eth 0/3 description Verizon Connection ip address 192.168.18.28 255.255.255.0 no ip proxy-arp ip access-policy "Secondary WAN" media-gateway ip primary qos-policy out VOIP no shutdown ! route-map Failover permit 1 match ip address Failover-ACL set ip next-hop 172.31.4.1 set interface null 0 ! ip route 0.0.0.0 0.0.0.0 172.31.4.1 track Failover ip route 0.0.0.0 0.0.0.0 192.168.18.1 50 ! . . . AdTranVoIP#
Next Steps
The customer had the AdTran consultant update the probe source to use the physical address of the AdTran router.
During the next maintenance window, the customer reconnected the AdTran, but the AdTran consultant was not able to ping the primary interface of the AdTran router. Based on the show mac address-table information on the 2960X-ISP and our revised diagram, we convinced the customer to update the VLAN placement of the AdTran interfaces on the 2960X-ISP.
Afterwards, the AdTran consultant could ping the physical address of primary interface 172.31.4.53.
After waiting for 30 minutes or so for lurking problems but finding no issue, we surmised that after resolving the AdTran probe source IP and placing the AdTran interfaces in the correct VLANs on the 2960X-ISP switch, the AdTran router could be successfully added to the network without bringing down the ISP circuits. Success!
Our best guess is that previously the incorrect probe address of 172.31.4.54 was being proxied by the ISP router, and / or that the ASA objected to seeing echo reply packets sent to his secondary IP address without ever sending an echo request.
Summary:
Check the diagrams and assumptions of the customer. When possible, review the configuration of any new devices that cause the network failures. Also, do not allow probes to use IP addresses not assigned to the device especially if they overlap elsewhere in the network.