Making ACI Configurations Consistent
This posting is a small update to the Cisco ACI section in my previous blog titled About Network Design Principles. How can one achieve consistency of ACI configuration?
Why is this a concern? In doing network assessments, I frequently compare configurations. I use a “golden config” and NotePad++ (Windows) or DiffMerge (Mac) for side-by-side comparison of configurations. Are the distribution switches in one place configured the same as those in another building or site, modulo addresses and things that have to differ? I often find differences, “entropy”. Configurations seem to drift a bit over time. Sometimes, key configuration elements are missing. Don’t people deploy from a template? And maybe periodically (annually?) check for config drift? Clearly, that doesn’t happen.
I hate to sound like an old CLI guy, but side-by-side diff tools spot that sort of thing pretty easily. How do you do that with a GUI?
And yes, checking e.g. interface configuration blocks for consistency is a bit harder. I have some code I wrote that can somewhat do that — take a template with wildcards, check it against matching blocks in a Cisco config for extra or missing lines. It really helped me spot a fair number of inconsistencies in some aging 6500 switches’ QoS, as an example. You could find or write code to do that with an ACI CLI dump or JSON dump.
The potential issue I’ve seen with ACI configured via GUI (especially in the presence of winging it / lack of planning) is that there are a lot of options, and sometimes several ways to do things. Give two engineers a task to do and they’ll find three different ways to do it!
For instance, if I need a private connection between the inside of a load balancer and certain servers or VMs, do I use a L2-only VLAN / bridge domain (which might be the way one does that with a physical switch), or do I do it via a contract? Does the server or VM have two interfaces (external / SLB facing and inside-trusted), or just one? I’m coming to prefer the latter, but that lead to a more routing / ACL approach prior to ACI, and in ACI, perhaps a contract-based approach.
The other potential issue I see is that the ACI GUI doesn’t exactly help identify all the factors that might impact a tenant, VRF, or bridge domain.
Both of those makes reverse-engineering an ACI implementation challenging (even with some automation to help). Documentation is usually missing, since people feel the GUI is the documentation (Something I sharply disagree with! To me the GUI is like using a microscope to examine a medical patient: too much detail, need the big picture first).
My conclusion: if one has standardized ways to do certain things in ACI, then there will be less to document on a case-by-case basis.
What can we do to better document what was built in in ACI? What can we build in ACI that makes it easier to document, or somewhat self-documenting?
Shifting gears for a second, I find it … interesting … in reading some recent NSX documentation, that NSX gets you away from having to think about traffic flows and device interfaces, and that you can write pure policy (my words). Coming from a Cisco firewall or access list (ACL) perspective, knowing the topology and how segment A gets to B allows writing targeted simpler policy. Maybe that’s a matter of taste. I’ve personally somewhat preferred taking flows into account so as to allow a more localized approach to firewall rules. Granted, if firewalling moves to endpoints, we’re probably talking about global policy.
I have the same concern with ACI — contracts can allow anything to talk to anything. How do you approach that to keep it manageable? Naming standards and other standards seem like one approach. Using visible “plumbing” (VLANs / subnets and routing) for “natural” connectivity rather than using contracts might help as well.
That brings us to leveraging Tenants and VRFs. If your contracts are primarily restricted to within-Tenant and / or within-VRF conversations, that helps narrow scope. Use naming to help keep that clear (what the Contracts are associated with). Then, at a minimum, have a clear naming for any contracts that connect VRFs or tenants to Common services or to other VRFs and tenants — or maybe use a firewall for that.
In short, impose some structure so that there are implicit boundaries and modularity, rather than arbitrary connection patterns. Divide and conquer, so that it is easier to figure out what might affect one tenant or VRF, and what is not relevant.
Of course, when you have standards, enforcing them or spotting deviations from them is the next challenge. I don’t have a great solution to that, other than maybe API-crawling scripts.
What brought this to mind is a recent blog on ipspace.net, by Andrea Dainese, Operating Cisco ACI the Right Way. Highly recommended reading!
Key items that resonated for me (my wording):
I’ll note that I still haven’t found anything online combining standard ACI design approaches with real world use cases. The Cisco ACI Best Practices Guide goes part-way, but just didn’t really come close to what I was looking for. It is still more of a “how-to-configure-and-build-it” guide providing basic tools. Then again, maybe I should adjust my expectations — Cisco provides the tools, it is up to us to figure out how to build something with them.
I did recently come across a white paper about Cisco IT’s ACI practices. That helps some.
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
Hashtags: #CiscoChampion #TechFieldDay #TheNetCraftsmenWay #ACI #NetworkConfiguration
Did you know that NetCraftsmen does network /datacenter / security / collaboration design / design review? Or that we have deep UC&C experts on staff, including @ucguerilla? For more information, contact us at email@example.com.
Making ACI Configurations Consistent
Six Tips to Help with Your Next Configuration Audit
Does Security Belong Near Endpoints?
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.