Amazon Transit VPCs and Transit Gateway
Something a bit different came up in troubleshooting the past week, something I’d like to share briefly in case it bites you.
Symptom: High CPU on core Nexus 7K in datacenter, leading to UDLD and VPC failure and some short-lived STP oddness. Nothing really showed up as hitting against the COPP (Control Plane Policing) policy. The site was running Strict level COPP, albeit from a prior NX-OS version. The newer Strict COPP template policy hadn’t been applied, but isn’t much different. Intermittent CPU spikes visible in “show process CPU history” command, which appear to have been normal “platform” process idle time processing.
More Data: Syslog showed some interesting messages. It showed a lot of Glean syslog messages like the following (carefully edited to hide identity):
%ADJMGR-3-IP_GLEAN_EXCESS_TRAFFIC: adjmgr  Excess traffic (10048 packets / 14709540 octets) were dropped in 00:04:48 seconds for ungleaned prefix .../32 on interface ... in vrf default
Cause: After tracking down the destination host (/32 in the syslog message), looking at the sources from NetFlow data, Googling, etc., someone realized that the destination was a NetFlow collector that had been recently decommissioned.
Resolution: Taking out/updating the NetFlow export commands on about 30 routers stopped the above syslog messages, and appears to have lightened the CPU load somewhat. Looked at in a different way, the Glean handling may indeed have been doing its job, protecting the CPU.
Explanation: When a packet has to be forwarded to a destination, the last hop router is responsible for ARP lookup of the destination MAC address. When hardware forwarding doesn’t find an adjacency for a known ARP entry, a Glean occurs so the supervisor can send an ARP to get the missing info. The supervisor creates a negative ARP entry until an ARP reply is received or a timeout period has passed. That saves pointless further ARP queries when one is already in progress, protecting the supervisor from a high load.
Note that Gleans occur in response to a normal packet being received with normal destination on a local subnet and requiring ARP, thereby triggering CPU activity. COPP only rate limits traffic going to the router control plane. The ARP table misses may not be handled by COPP, although apparently the Nexus 7K has hardware Glean rate limited.
Gratuitous Advice: When decommissioning a NetFlow collector, particularly one with lots of source routers feeding it, remove the relevant NetFlow (or IPFIX) export commands first. That way, you avoid what might be described as a “self-inflicted DDoS attack.”
I do have to note that the transient VPC/STP issue root cause is still unclear. It was a one-time transient condition. Some spike in NetFlow traffic might possibly have bumped up the Gleans to where they were a problem. Since the transient, we haven’t seen anything peg the CPU nor cause other problems. So there might well have been another cause for the VPC problem, or another cause compounded by the Glean situation.
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
Amazon Transit VPCs and Transit Gateway
Aruba Wireless Controllers: Architecture & Configurations
Dealing with Performance Brownouts
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.