This posting is a small update to the Cisco ACI section in my previous blog titled About Network Design Principles. How can one achieve consistency of ACI configuration?
Why is this a concern? In doing network assessments, I frequently compare configurations. I use a “golden config” and NotePad++ (Windows) or DiffMerge (Mac) for side-by-side comparison of configurations. Are the distribution switches in one place configured the same as those in another building or site, modulo addresses and things that have to differ? I often find differences, “entropy”. Configurations seem to drift a bit over time. Sometimes, key configuration elements are missing. Don’t people deploy from a template? And maybe periodically (annually?) check for config drift? Clearly, that doesn’t happen.
I hate to sound like an old CLI guy, but side-by-side diff tools spot that sort of thing pretty easily. How do you do that with a GUI?
And yes, checking e.g. interface configuration blocks for consistency is a bit harder. I have some code I wrote that can somewhat do that — take a template with wildcards, check it against matching blocks in a Cisco config for extra or missing lines. It really helped me spot a fair number of inconsistencies in some aging 6500 switches’ QoS, as an example. You could find or write code to do that with an ACI CLI dump or JSON dump.
ACI Variations on a Theme
The potential issue I’ve seen with ACI configured via GUI (especially in the presence of winging it / lack of planning) is that there are a lot of options, and sometimes several ways to do things. Give two engineers a task to do and they’ll find three different ways to do it!
For instance, if I need a private connection between the inside of a load balancer and certain servers or VMs, do I use a L2-only VLAN / bridge domain (which might be the way one does that with a physical switch), or do I do it via a contract? Does the server or VM have two interfaces (external / SLB facing and inside-trusted), or just one? I’m coming to prefer the latter, but that lead to a more routing / ACL approach prior to ACI, and in ACI, perhaps a contract-based approach.
The other potential issue I see is that the ACI GUI doesn’t exactly help identify all the factors that might impact a tenant, VRF, or bridge domain.
Both of those makes reverse-engineering an ACI implementation challenging (even with some automation to help). Documentation is usually missing, since people feel the GUI is the documentation (Something I sharply disagree with! To me the GUI is like using a microscope to examine a medical patient: too much detail, need the big picture first).
My conclusion: if one has standardized ways to do certain things in ACI, then there will be less to document on a case-by-case basis.
What can we do to better document what was built in in ACI? What can we build in ACI that makes it easier to document, or somewhat self-documenting?
Shifting gears for a second, I find it … interesting … in reading some recent NSX documentation, that NSX gets you away from having to think about traffic flows and device interfaces, and that you can write pure policy (my words). Coming from a Cisco firewall or access list (ACL) perspective, knowing the topology and how segment A gets to B allows writing targeted simpler policy. Maybe that’s a matter of taste. I’ve personally somewhat preferred taking flows into account so as to allow a more localized approach to firewall rules. Granted, if firewalling moves to endpoints, we’re probably talking about global policy.
I have the same concern with ACI — contracts can allow anything to talk to anything. How do you approach that to keep it manageable? Naming standards and other standards seem like one approach. Using visible “plumbing” (VLANs / subnets and routing) for “natural” connectivity rather than using contracts might help as well.
That brings us to leveraging Tenants and VRFs. If your contracts are primarily restricted to within-Tenant and / or within-VRF conversations, that helps narrow scope. Use naming to help keep that clear (what the Contracts are associated with). Then, at a minimum, have a clear naming for any contracts that connect VRFs or tenants to Common services or to other VRFs and tenants — or maybe use a firewall for that.
In short, impose some structure so that there are implicit boundaries and modularity, rather than arbitrary connection patterns. Divide and conquer, so that it is easier to figure out what might affect one tenant or VRF, and what is not relevant.
Of course, when you have standards, enforcing them or spotting deviations from them is the next challenge. I don’t have a great solution to that, other than maybe API-crawling scripts.
Use the API, Luke
What brought this to mind is a recent blog on ipspace.net, by Andrea Dainese, Operating Cisco ACI the Right Way. Highly recommended reading!
Key items that resonated for me (my wording):
- How do you get a team to configure things the same way, given all the configuration options? Answer: API, automation.
- How do you get a team to assemble components quickly and in a standard way? I take this as a wish for a common standard approach to implementing various things. Answer: automation, API.
- The part about detecting mistakes. With GUI, how do you do anything comparable?
I’ll note that I still haven’t found anything online combining standard ACI design approaches with real world use cases. The Cisco ACI Best Practices Guide goes part-way, but just didn’t really come close to what I was looking for. It is still more of a “how-to-configure-and-build-it” guide providing basic tools. Then again, maybe I should adjust my expectations — Cisco provides the tools, it is up to us to figure out how to build something with them.
I did recently come across a white paper about Cisco IT’s ACI practices. That helps some.
References
- About Network Design Principles: https://netcraftsmen.com/about-network-design-principles/
- Operating Cisco ACI the Right Way: https://blog.ipspace.net/2019/02/operating-cisco-aci-right-way.html
- Cisco ACI Best Practices Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices.html
- Cisco IT ACI Design: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Design.pdf
Comments
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
—————-
Hashtags: #CiscoChampion #TechFieldDay #TheNetCraftsmenWay #ACI #NetworkConfiguration
Twitter: @pjwelcher
NetCraftsmen Services
Did you know that NetCraftsmen does network /datacenter / security / collaboration design / design review? Or that we have deep UC&C experts on staff, including @ucguerilla? For more information, contact us at info@ncm2020.ainsleystaging.com.