I think I have a love / hate relationship going with the Cisco 6500 Sup720-10G module ports. On the one hand, they’re darn handy and cost-effective. On the other hand, they represent one more set of one-off gotchas that make the whole 6500 complex. I won’t say unnecessarily complex, but definitely taxing to track the idiosynracies of, the capabilities and limitations of the various modules, etc. Bleeding edge technology (or what once was), cost effective, however it seems like it could all be a lot more user friendly. (Take QoS for example … but that’s another rant.)
The purpose of this blog is to pass along a couple of quirks that you might encounter. They’re adequately documented, but you may not yet have read the appropriate part of TFM (The Fine Manual).
Using Those 10G Ports
First, the good news. If you’re using a Sup720-10G in a closet switch, hey, you’ve got 2 10G uplinks right there. If you’re doing High Availability closets with dual Sups for IP Telephony, well, then you’ve got 4 10G uplinks, good for 2 x 20 G EtherChannels. So far so good.
What about in the data center? If you have dual switches for servers to dual home to … not quite so good. You can use one of the ports for 10 G cross-link to the other switch, and have one left for 10G uplink. That’s OK but not great if you’re like me: I like dual uplinks from each server switch to each distribution layer switch, whether I’m doing a L2 or a L3 design. So you have to add a 6708 or 6716 blade, to get a couple more 10G ports to do that with. Or add a second Sup to each switch — which is less common in data centers, two chassis generally being adequate redundancy.
One alternative that comes to mind is to VSS the two server switches. You can’t currently do that with dual Sups. So you might use one of the 10G ports for VSL, and one for uplink. However, Best Practice is to have a second VSL port in the EtherChannel on another module. So you end up adding a 6708 blade, using one port to beef up the VSL link, and maybe another to get to 2 x 10G EtherChannel to each distribution switch. (Or more.) That drives up the cost per chassis a bit.
So that’s not terrible, it just leads me to the conclusion that the two 10G ports aren’t quite enough. And until you start seeing 10G NICs on servers, most of those additional 10G ports are … less than useful.
QoS and EtherChannel Using the Sup720 10G Ports
The next little surprise comes when you try to use the 10G ports on the Sup720-10G for EtherChannel. Generally, we build EtherChannels using ports from different modules (blades) so the logical link survives a module failure. Well, the ports on the Sup720 aren’t the same as those on the 6708 module, QoS-wise. And so the EtherChannel of a 10G port on say blade 1 and one on the Sup720 won’t form.
No big problem, it’s Cisco IOS, there’s a command for that! The command is “mls qos channel-consistency“. For details, see http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.1E/native/command/reference/M1.html#wp1234645. After some thought, I decided I don’t think I care that the EtherChannel ports are doing slightly different QoS queueing. As long as DSCP is recognized, and priority queuing used where appropriate, I doubt I’m going to be able to see effects from the slight differences in QoS handling. [If you suspect otherwise, please let me know, or add a comment.] The reference above actually says the command allows mixing of ports with and without strict priority queues, however I sure have the impression that both ports I’ve worked with supported priority queues, just different numbers of queues and thresholds.
VSS, VSL, and QoS on the Sup720 10G Ports
When you configure VSS, you discover the VSL has some ramifications. Specifically, the early code only supports trust of COS. I generally do DSCP, since COS is less finely granular, Cisco supports DSCP even on L2 links, and DSCP can then be “universal” except on very limited L2 / COS-only switches.
In addition, putting a port into the VSL programs the ASIC — which means you get the same limited QoS on the other 10G port on the Sup720, whether or not it is part of the VSL. Furthermore, the VSL QoS settings prevent you from applying queuing configuration or a service policy to the ports in question. So if you planned on using the second 10G port on the Sup720-10G as an uplink / downlink or as part of an EtherChannel, well, you can do that, but possibly not with the QoS that you might otherwise use.
For details, see http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/vss.html#wp1060412. (And thanks to our Keith Gardiner for researching this and doing some lab work to confirm behavior.) This reference appears slightly incorrect in that it talks about VSL configuring all ports on the same ASIC. I now believe that is only true for the Sup720-10G blade, not for other modules. Keith’s testing showed that even with a port on a 6708 module in the VSL, we could apply our standard QoS settings to the other ports on that blade.
The above reference does provide an alternative: configure “mls qos 10g-only“, disabling the Gig ports on the Sup720-10G blade. Unfortunately, we had previously decided to use the copper 5/3 port for universal network admin access, with a locally configured /30 DHCP scope. (You know, when you’re in a closet or the data center and you need a port to connect to the Internet or some internal server to check some documentation.) And the context we’re in requires a couple of SFP fiber ports for fiber-attached (old) servers — we’d rather use the Sup720-10G ports than add a blade to support a couple of such server attachments.
Summary
The ports on the Sup720-10G are rather handy (and cost-effective). But they do have their quirks. And it is useful to know them ahead of time. I can understand the Cisco engineers and programmers being limited to what the ASIC could do, and to having to set programming priorities. Nonetheless, it would be nice to have a summary of such quirks, limitations, and interactions provided before you are in the lab or production and encounter them “the hard way”. I hope this blog helps in that regard.
Note added 2/10/10:
There is a great VSS design resource available at http://www.ciscosystems.ch/en/US/docs/solutions/Enterprise/Campus/VSS30dg/campusVSS_DG.html.
Hi, it seems that you are referring to a U-shaped design between the Server Farm Access and the Distribution Layer since you refer to a crosslink between the server farm access switches. My question is that on quite a few occasions I have got the impression from cisco that they prefer a V-shaped design between the Access and Distribution layer even if it involves more than 1 access switch which is traditionally the case in the server farm access layer design. I personally prefer the U-shaped design so just wanted some feedback and suggestions on why do you prefer the U-shaped or if you do not than why not. Thx for all your help.
In most of my recent designs, the access switches are routing from the server VLANs up to the distribution layer. So in order to dual-home any servers, each VLAN must be present on both access switches, hence the L2 crosslink.
For the upstream connection, I generally follow Cisco’s "bowtie" or "triangles" recommendation, and provide an uplink from each access switch to each distribution switch. I even do that with VSS (with cross-chassis EtherChannel) for chassis to upstream survivability should one upstream switch fail. For routed connections you want point-to-point interfaces for rapid link failure detection. As soon as you throw VLANs and multiple ports into the mix, your routing convergence will be slower.
For larger VLANs, then one does pretty much have to run VLANs up to the distribution layer. I lean towards VSS for that, minimizing the use of spanning tree protocol, and using all uplinks. The problem with the L3 approach, the U or inverted-U is that if you lose the cross-link, you’ve then got a discontiguous VLAN at the access layer. With a L2 access layer and bowtie, you’ve got connectivity to the other access switch via both distribution switches. Using no more ports than an uplink and an access crosslink would.
Hope that answers your question.