Designing for Cisco Nexus 9K
I’m writing this at and after the Cisco / Insieme #ACI (Application Centric Infrastructure) launch today in New York City. There’s much to cover! Recall that in some prior blogs I threw out a bunch of guesses in a prior blog, and a couple of things I hoped for. I’m amused that some of them appear to actually be part of the announcements, or unnannounced intent. The announcements today were broad in coverage and to some extent short in detail. In particular, ACI details are still emerging.
This will be a two part (at least) blog, with this one focusing on the hardware / optics side of announcements. It is already long enough, just covering that material. I hope to garner a bit more ACI information today and follow up with what I can in a Part 2 soon, focusing on ACI.
As more comes out, we may all be amazed at the breadth of the vision. Some of the key Insieme personnel are … visionary, intense! With UCS and now perhaps ACI, Cisco may be demonstrating that there is virtue in allowing a new technology to settle a little, to have more time to analyze what is key in a market or technology. They have been participating: OnePK to provide agility, ability to change direction quickly, OpenDaylight for cloud automation support, 1000v as early service-chaining and to build skills working with the lead hypervisors and learn how to provide consistent functionality across hypervisors. And it appears that ACI now their attempt to define the market and stake out a position, with much more vision that the innovation around the edges that some other network device vendors have produced.
My wishlist that I’m referring to includes:
A good starting point is what was announced. I was a bit surprised there wasn’t any one summary presented listing all the new stuff! The announcements are definitely a big deal, a bid to manage and automate the whole datacenter and cloud, from network to server, application, storage, and security, encompassing both physical and virtual. I’ve read the various blogs to try to pull together a comprehensive list.
Here’s my own list of the major announcement items, culled from various sources:
For additional information, unfiltered through my brain, see also the following URLs:
Let’s look at some of the most noteworthy details, easy ones first…
Nexus 9508 takes up to 6 fabric modules at 5.12 Tbps each forwarding, directly connected to line cards.
The three line cards available are:
This switch can be a big edge switch, leveraging N2K FEX. Or an aggregation layer switch. It seems positioned as a much faster 6500 replacement in these roles. Another use case is leaf or spine in a leaf-spine CLOS tree, possibly with the 9300 platform as leaf nodes. At least they didn’t call that “SPLINE” the way Arista did. (The TFD crew seems to not like that term.)
A diagram shows up to 12 spine 9500 switches, and claims support for scaling to 200,000 x 10 G servers or more. (With FEX and some FEX oversubscription, I imagine.)
One NX-OS binary supports the N9K switches. NX-OS Plus is required to support the ASIC controller.
See the datasheet (link above) for details. And you can play the game, “here’s the impressive long list of features, but what did they leave OUT?”
The Nexus 9300 is a ToR switch, with 2 models:
The Nexus 9300 also supports N2K FEX.
For those paying attention, the N9K series supports VXLAN. And 100 G line cards are coming, with adequate fabric capacity to support them.
One aspect not prominent in the announcements was positioning and pricing. How are these different than the Nexus 7K/5K in terms of features? If the price per port is lower, is there some displacement of the earlier Nexus models (“virtual end of life”) coming? And while one can run them using NX-OS, the ACI chipset and central APIC controller mode is the really interesting alternative.
The N9K is a mix of merchant silicon and customer ACI logic (per line card).
The Cisco QSFP 40 Gbps BiDirectional (“BiDi”) transceiver uses duplex LC connectors for 40 G over MMF at short distances. The impact is that you can use your 10 G MMF fiber plant in the datacenter for 40 G connections, as is, at distances up to 100 meters over OM3 fiber. The cost is apparently about twice that of a short distance 10 G SFP+. The transceiver apparently uses two wavelengths on each strand (one for each direction) forming two 20 Gbps channels.
The Cisco Application Virtual Switch (AVS) isn’t hardware, but appears more straightforward to explain than ACI and profiles. Its product page is http://www.cisco.com/en/US/products/ps13463/index.html.
“Based on the highly successful Cisco Nexus 1000V virtual switch, AVS provides feature support for the ACI application policy model, full switching capabilities, and more advanced telemetry features.”
The role of AVS appears to be a 1000v footprint for each hypervisor, that ties it back into the Cisco ACI system. Part 2 will cover the telemetry aspect.
One item that caught my eye: “Compared to other hypervisor-based virtual switches, AVS provides cross-consistency in features, management, and control through Application Policy Infrastructure Center (APIC), rather than through hypervisor-specific management stations. As a key component of the overall ACI framework, AVS allows for intelligent policy enforcement and optimal traffic steering for virtual applications.”
That is, AVS provides a hypervisor-neutral set of functionality. That seems like it may appeal to those trying to avoid being locked into one hypervisor vendor, and looking for cloud portability perhaps as well. (From what I hear, VMware pricing has incented many customers to hedge their bets for fear of hypervisor lock-in subjecting them to future licensing fee shock.)
The trip to NYC for the ACI Launch provided quite a bit of fun and intellectual stimulation. I’d like to thank Cisco, especially Amy Lewis (@CommsNinja), and Tech Field Day in the person of Tom Hollingsworth (@networkingnerd) (http://networkingnerd.net/), for inviting me! And the rest of the TFD folks, for the interesting discussions!
Designing for Cisco Nexus 9K
Disaster Recovery and Lead Time: Are You Prepared?
Network Monitoring: Top Performance Items to Watch, Part 2