Learn to drive teamwork and profitability through unified communications in Cisco’s new eBook. Download now!

I recently prepared a vPC Best Practices Checklist, a list I and a consulting customer could use as a vPC configuration planning / review checklist. The information is a mix of a few ideas of my own combined with items pulled from several Cisco documents about vPC. I’m hoping you may find this checklist useful. Yes it is terse: it is supposed to be a checklist, and I tried to keep it short. (Something I can find challenging.)

List of vPC Best Practices

  • You cannnot build a vPC where one end of the port channel is split across two VDC’s of one Nexus switch.
  • The vPC domain ID should be unique within its L2 domain. If OTV is contemplated, the  vPC domain ID should be unique across all datacenters.
  • The vPC role priority should be configured so the primary device is known. Lower priority wins. Default is 32667. Knowing the primary is useful when troubleshooting, also so you know where to home singly-attached devices (next item). Generally this switch has a name ending in -01 or -A.
  • (NetCraftsmen) Configure the role priority explicitly to non-default values on both switches to make your intent clear to those who may not know the default value. (Admittedly, they may also not know smaller number wins.)
  • Singly-attached devices should be connected to the vPC primary device.
  • (NetCraftsmen) The peer-switch command should be configured in vpc domain sub-mode to make the peers look like one switch from the STP perspective. (This is flagged as “NetCraftsmen” since Cisco doesn’t quite explicitly recommend it as a best practice, nor discuss possible drawbacks.) The peer-switch command (in vpc domain mode) apparently improves STP convergence if the vPC primary fails.
  • The vPC peer link should be robust. A recommended practice is for the peer-link port-channel to contain links off two separate modules if possible. If not, using a track command (tracking upstream interfaces  on the same module) is recommended to avoid black-holing traffic on lower speed server links if that one 10 Gbps module fails.
  • For vPC member links, make the configured vPC number match the port-channel number: that way there is one less number to track or to have to discover when troubleshooting.
  • Configure all the port channels in the vPC using LACP with the interfaces in LACP active mode.
  • Downstream STP (Spanning Tree Protocol) should be left enabled, as insurance. In case of a double failure affecting both the vPC peer-keepalive and peer links, STP is needed.
  • The STP root and HSRP active configuration should align with the primary vPC peer. This is no longer strictly necessary due to improved spoofing by the secondary, but is still useful for troubleshooting.
  • Understand that both vPC peers will forward traffic sent to the HSRP vMAC (instant load balancing).
  • Bridge Assurance should be configured on the vPC peer link ends. Entering the vPC peer link command does this automatically.  ISSU requires having no Bridge Assurance on vPC member links.
  • Best practice is to not configure the member links with Bridge Assurance: doing so can apparently hamper STP convergence in some situations.
  • The vPC peer-keepalive path should be robust, and should use a port other than the vPC peer link. While vPC will continue to operate if the peer keepalive address is not reachable, it will not be able to re-form the vPC peer-link after an outage.
  • One way to increase keepalive robustness with a separate routed physical link for keepalives is to put the keepalive port into a dedicated VRF, so that dynamic routing cannot mis-direct traffic being routed to the vPC keepalive peer address.
  • Do not mix routing and vPC. There must be no routing peering over vPC member links to SVI’s on the vPC peers. That means that SVI’s for a VLAN should be either on the vPC peers or the attached access switches, but not both. Preferably the SVI’s should only be on the vPC peers, to reduce the risk of errors.
  • (NetCraftmen) Make all vPC member-link VLANs passive for routing, then you won’t have inadventent adjacency problems. This also avoids large numbers of pointless routing adjacencies for the vPC peers (pointless because you almost certainly do not want failover transit via an access switch)..
  • It can be useful to put in place a L3 cross-link between each distribution layer VDC pair. These might be set up as two-link port-channels, using ports on two different line cards for robustness. Point-to-point VLANs might be used for routing adjacency between the distribution switches.
  • Any non-vPC VLANs should be not allowed on the vPC peer-link trunk, instead trunked between vPC peers over this separate non-vPC-peer port-channel cross-link.
  • If you want your vPC peers to form routing adjacency over the vPC peer link, do so in a VLAN that is different than all vPC member VLANs, to avoid surprises and problems. Better: make all vPC member link VLANs passive for routing, and do all routing on separate links, both to vPC member devices and to vPC peers.
  • There is no need for aggressive FHRP timers with vPC, since both peers forward. This behavior also provides HSRP load-balancing without needing to switch to GLBP.
  • If you configure static MAC addresses, they must be configured identically into both vPC peers.
  • Enable the “ip arp synchronize” and “ipv6 nd synchronize” commands to support faster convergence of address tables between the vPC peers.
  • If you’re really organized and want a domain-id numbering convention, bear in mind that most vPC usage will be mostly server to access switch, with some access switch to aggregation. Aggregation to core is usually L3, for which vPC is not suitable.

vPC Special Situations

  • If you are having problems with EMC or NetApp NAS servers, consider using the “peer-gateway” command, to allow forwarding of traffic sent to the BIA MAC of the peer. This command is intended to address problems with certain EMC and NetApp filing systems, which do not do ARP but instead use the BIA MAC source address from received traffic. Using this command may appear to fix unicast routing over vPC issues, however, the topology is not supported and apparently has known problems with IPmc (and so should be avoided if at all possible). [NOTE: Cisco documentation recently is mixed on this topic. It is hard to tell if some people didn’t get the memo saying “unsupported”, or perhaps the official position on this changed recently.]
  • A vPC peer may wait a long time for its peer after a dual reboot. Configure auto-recovery “auto-recovery [reload-delay time]”. This is used when both vPC peers are rebooted, so that the secondary does not wait forever for the primary to come up before assuming it is the only active peer.
  • Consider configuring the interface delay restore timer as well. This causes the vPC to stay down after a reboot until links can come up and routing protocols converge, to avoid black-holing traffic.
  • The following commands are not supported in vPC mode: “ip pim spt-threshold infinity” and “ip pim use-shared-tree-only“. (As of 12/2011.) NetCraftsmen usually recommends configuring the SPT threshold to infinity to reduce IPmc state, particularly if IPmc servers and Rendezvous Points are in the datacenter(s). This is not yet possible on a device with vPC’s.

Fairly New vPC Capabilities

You should know about the following for your design toolkit:

  • vPC+: When you’re running FabricPath, allows load-balancing to an edge pair doing downstream vPC.
  • Enhanced vPC: 5.2.3 / 6.0 code allows you to dual-home N2K’s to N5K’s via vPC, and now additionally run vPC to servers attached to those N2K’s. Neat stuff!

Reference Links

vPC Best Practices documents:

vPC and routing:

vPC+ and FabricPath:

Enhanced vPC (and nice diagrams of it):

Peter Welcher

Peter Welcher

Architect, Operations Technical Advisor

A principal consultant with broad knowledge and experience in high-end routing and network design, as well as data centers, Pete has provided design advice and done assessments of a wide variety of networks. CCIE #1773, CCDP, CCSI (#94014)

  • Pete Welcher

    Thanks for good comment and links: James points out that [b]UDLD aggressive is not recommended on member links either[/b].

    He provided the following useful links on the topic:

    [url]http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html#wp1037426[/url]

    [url]http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsz76892[/url]

    [url]http://www.scribd.com/doc/77905734/Nexus7000-vPC-Best-Practices-and-Design-External-CollectedPPT[/url]

  • Jen Shen

    Great check-list you made there. Awesome ;D

  • Robert R

    Do you recommend the use of port channel numbering schemes that show the vlans that they carry such as Po1500 carrying vlan 1500 & 1501 then Po1502 carrying vlan 1502 & 1503 & 1504 then Po1505 carrying vlan 1505,1506,1507,1508 ?
    Or should there be defined functional ranges such as
    "5K/UCS -> Po500-550",
    "Local closets -> Po600-650",
    "Remote Buildings -> Po700-900" ?

    Thank you,

  • Pete Welcher

    In general, I don’t think you can do that, the port channels may well carry a lot of VLANs. In the special case where you’re going to a VMware host, maybe you could tie 3 VLANs for console/mgmt, production, and VMotion to the vPC to that host. Or 6 or more. For a blade chassis, I’d be inclined to do blocks of 8 or 16 VLANs to allow for growth. I wouldn’t expect that to last long. Most sites don’t manage to be so organized, and the whole point to the "flat L2 world" I keep reading about is that server admin entropy drives randomization of VLAN locality (i.e. you end up needing every VLAN everywhere).

    Where strong security is desired, I’d hope people would be a bit careful about the mix of VLAN’s and VM’s on VMware … but that’s another topic.

  • Pete Welcher

    While reviewing the Nexus DCUFI course materials, I ran across: don’t mix vPC peer-link endpoints across both types of linecards: M1 and F1 modules. Your vPC peer-link should run only on M1 series linecard ports or F1 ports.

  • Will

    Just found this website. Love your posts.

  • Gonzalo Diaz

    Nice article, Cisco Documentation is very ambigous in certain aspects. Hope you can add this explaination from Cisco, that explains why you shouldn´t mix vPC with routing using the peer link. I’m currently adding a L3 dedicated link to overcome certain issues with AS400 machines living in vPC under 5K Aggregation Pair.

    This is from Cisco:

    “The Nexus 7000 will
    drop any packet that comes in on a vPC etherchannel, traverses the vPC peer
    link, and then tries to leave via a vPC etherchannel (including etherchannels
    that do not go back to the original device but are still in the same layer 2
    domain). This is the designed layer 2 loop prevention mechanism for the Nexus
    7000 as it does not block ports for vPC VLANs.”

    Regards

  • See some of the Cisco docs re this issue with PAgP negotiation vs. hard-coding port-channel on. Basically, if the two ends of the port-channel don’t negotiate, any error can result in one end doing port-channeling and the other not. That’s a guaranteed STP loop. I’ve seen it bring down a pretty good sized datacenter when an 8th pair of 6509 access switches were added and someone forgot to group the 4 x 1 G ports on the access switch end (with older port-channel, but the principle still applies). So I think of LACP active as insurance against accidents causing a STP loop and taking the datacenter down.

View more Posts