Configuring the Customer Side of an MPLS VPN WAN, Part 1

I’ve recently been doing a lot of consulting work involving variations on a common theme. The general theme is how do I configure my routers to work with the MPLS VPN service I have bought or am about to buy from Provider X. If the service is to be your sole WAN connection, there’s little problem or complexity. Do get the carrier to hand routes off to you in your IGP, unless you happen to want EBGP in your life. If you are going to have two WAN connections, well, things can get more interesting. Which is what this article is about.

I’ve been noticing how the requests for MPLS WAN related consulting work has evolved over the years:

• Initially, folks discovered there is some “interesting” routing behavior when you tie an existing IPsec VPN or legacy WAN as alternative / backup path to a shiny new MPLS VPN.
• More recently, sites are doing two-carrier MPLS (i.e. diverse connections via two separate MPLS VPN clouds), often with MPLS support for QoS
• Layer 2 MPLS VPN has come onto the scene, often with QoS as well
• And some sites are now doing the legacy WAN + MPLS VPN approach, with the new MPLS VPN as a quick fix to support IP Telephony (IPT) or IP Video-conferencing (IPVC) and the older QoS-less and possibly congested WAN for data

I haven’t noticed much on the web about how to configure for these sorts of situations. I just googled as a check, and found a couple of sites, plus links to prior writings of mine.

The lack of discussion is a bit surprising to me, since the routing can get a little more complicated than your average internal routing does. I’ve been aware of that since around 2003, when I started consulting on this. (And in presenting at a Cisco Powered Network, got the impression that MPLS Providers seemed to be all thinking that they’d be your only WAN connection, and weren’t prepared for some of the issues discussed below.)

MPLS WAN Scenarios

I see this as three basic WAN settings or scenarios, each requiring mildly different approaches to configuration, and each entailing different important considerations. The settings are:

• Legacy WAN with IGP (EIGRP or OSPF) routing plus Layer 3 (routed) MPLS VPN WAN
• Dual-carrier L3 MPLS VPN WAN
• Legacy + Single, or dual-carrier L2 MPLS VPN

The last situation, differentially routing certain traffic over the MPLS WAN, with data over the legacy WAN, I see as QoS-related: how do I route traffic based on QoS needs or based on DSCP bit settings?

If you have IPT or IPVC or video, you probably do want an MPLS service supporting QoS via DSCP bits (not per-VLAN QoS, which I think is pretty useless). But that’s Yet Another Topic, more properly a QoS topic, and not much different than regular QoS. You just have to match the carrier’s DSCP bit preferences (if non-negotiable), etc., at the WAN edge.

This blog will cover the legacy + MPLS scenario, without or with the differential QoS over MPLS requirement. Others to be written will take a look at the other scenarios.

MPLS WAN: The Big Picture

If your area has fiber available (major league cities, etc.), you may find that Ethernet access to L2 or L3 MPLS VPN or to L2 VPLS or Ethernet Private Line (EPL) service is available and fairly cost effective, while providing flexibility as far as ability to rapidly add bandwidth as needed. In a recent set of preliminary quotes from vendors, we saw that there’s a bit of a fee for the fiber access, but the incremental costs as you scale up bandwidth are low. If your network is in the T1 range and you’re looking at going above that, the alternatives are pretty limited (IMA and ATM has hardware costs and generally only provides enough bandwidth for 1-2 years of use), so cost-effective Ethernet services with 1 or 5 Mbps increments is good news!

I’m starting to see more any more of a shift to the Ethernet-based access, which is simple and very useful. It does have the problem of “some sites left behind” — more rural sites may only be able to get T1 or costly T3 access still, while others are operating at 10, 100, or even 1 Gig speeds. This tends to result in apps that require more bandwidth and don’t work so well at the slower sites.

The big picture here is that MPLS VPN is great, it can be very cost-effective. Both L3 and L2 MPLS VPN have quirks that make them a bit different than more traditional WAN designs. Nothing terrible, but factors that do have to be taken into account.

For L3 MPLS VPN,the challenge is that your Customer Edge (CE) router exchanges routes with the Provider Edge (PE) router, and that’s just the way it works. Most providers seem to now grudgingly offer EIGRP, often telling sites you’ll be the first customer on it, or back-hauled to a POP with a Cisco MPLS edge router at higher latency. That full disclosure is good and honest, but does seem to have the (intended?) effect of driving customers to NOT using EIGRP for the CE-PE routing.

L2 MPLS VPN is attractive to those who want to do their own routing. I’ll note in passing that this is slighly illusory, in that routing in the MPLS core still underlies the L2 connectivity. On the other hand, the carrier should be doing some mix of SONET, MPLS Traffic Engineering (TE) Fast Re-route (FRR), fast core IP re-routing based on limited prefixes, tuned timers, etc. And it’s all just up / down from your perspective. The bigger gotcha with L2 MPLS VPN or VPLS is that it makes the “WAN cloud” act like a giant Ethernet switch. Routing is not really intended for 25 or more routers on a LAN, let alone 100’s. You can do that, but it’s not a good design.

Legacy + MPLS

Let’s get technical now, and talk about my first scenario, Legacy WAN plus Layer 3 MPLS VPN for WAN.

Generally the legacy WAN routers are talking EIGRP or OSPF to each other, an IGP (Interior Gateway Protocol). If your MPLS carrier provides EIGRP or OSPF handoff in the right form, that too will appear to be IGP routing and life is good (and simple to manage). Cisco has added features to carrier MPLS allowing the carrier MBGP to carry EIGRP or OSPF routing metrics and re-advertise them back to your CE router as internal routes. There are minor quirks to consider and work with your carrier on if you have other connections (“backdoor connections”) between your MPLS sites.

The main thing that go wrong with the above is for the carrier to mess up on the EIGRP AS matching, at one or more sites. That has the effect of making the routes learned on the MPLS side external EIGRP routes. EIGRP much prefers internal routes (administrative distance). So if that happens, your routing will be steering traffic over the legacy WAN rather than the new MPLS WAN.

The other way routing can be designed is to talk EBGP to the carrier. (I’m ignoring static routes, which are useless for dynamic re-routing when you have two connections — and all my scenarios have two connections.) If your CE-PE routing on the MPLS WAN is EBGP, there are two sub-cases.

Sub-Case 1: If you only have one router (e.g. small remote office), then it’s talking EIGRP or OSPF on one side, EBGP on the other. Due to administrative distance, EBGP wins, so the router will prefer the MPLS WAN for any prefix learned on that side. That’s usually what you want, so lucky you, things work as you’d want!

Sub-Case 2: You have two routers, one (usually the older one) talking to the legacy WAN, the other (sometimes shiny new) talking to the new L3 MPLS WAN. EIGRP for legacy, EBGP for MPLS.

How do the two routers exchange routes? Generally people bi-directionally redistribute the EBGP into EIGRP or OSPF. (Some details follow below.) If you do that, the MPLS router has an IGP and EBGP as route sources, and EBGP wins. So if it receives traffic, it will generally prefer the MPLS WAN. That’s good. On the other hand, the legacy router has internal IGP routes plus the external IGP routes learned from the MPLS router’s redistribution. For an IGP, internal is better than external, so the legacy router “wants” to send traffic over the legacy WAN. Not so good. So will any other routers or L3 switches at the site. Definitely not so good.

This is why I like doing EIGRP or OSPF as the CE-PE protocol, matching your IGP. It keeps things simpler.

From a business perspective, I strongly recommend that the CE-PE routing protocol handoff be a major vendor selection criterion.

What can you do about this? If you have no other L3 devices at the site, you can make the MPLS router the HSRP / VRRP primary, with WAN interface tracking, or even whether an important route, like default, is successfully being learned from the MPLS peer. That way, devices’ traffic will go to the MPLS router, it prefers the MPLS link, and life is good. If some site falls out of the MPLS cloud (so to speak), the prefix is only learned on the legacy side, so the MPLS router will redirect traffic to that site to the legacy router, which in turn will forward it out the legacy link.

Ah, but if you have other L3 devices at the site, then things are a bit more … interesting. Some ideas:

• You can reconfigure the legacy WAN to use EBGP or IBGP. Painful, but it’ll work. (Which to use, merits a separate discussion.)
• You can try altering administrative distances for external routes, carefully. That’s a great way to create a routing loop — and didn’t quite do what’s necessary, the last time I tried it with EIGRP in the lab. (It appears like it should be possible with newer code, if configured on all site routers — I just haven’t tested it in the lab yet.)
• You can use a “neighbor … distance” command, to bias the adjacent site router(s) to give high admin distance to routes from the legacy WAN router. Deeper into the site, it doesn’t matter, as long as the routers adjacent to the legacy router “prefer to” send traffic towards the MPLS router. I haven’t done this or tested this, but it seems a lot cleaner than changing the administrative distance for external routes at every site router.
• You can try doing different IGP processes for WAN and LAN, and redistributing between them, but that gets rather messy. Not recommended. Particularly as you’ll need bi-directional redistribution filtering, which adds a bit more complexity. (Route tags are your friend, if you’re doing this.)
• You can run “the other” IGP at the site, redistributing both the legacy WAN IGP and the EBGP into it. Thus if the legacy WAN uses EIGRP, use OSPF at the site. You’ll probably be redistributing bi-directionally, unless you want to configure a static site summary and redistribute that for advertisement across the WAN. This gets just as ugly as the previous idea.
• Yet another approach (with distinct negatives) is to tunnel between CE routers on the MPLS side, allowing you to run an IGP over the tunnels (GRE or IPsec, generally). If you’re running IPsec VPN between CE devices for extra security, then you’re set — leverage that for routing.
• You can run EBGP or IBGP on your legacy WAN, but that means re-doing all your legacy WAN routing, which is painful.

Adding QoS Preferential Routing to the Mix

Actually, this case is simpler. If you only have one site router, you can just do metric offsets or Policy-Based Routing (PBR) on it. We’ve done this sort of thing for PBX to PBX IPT traffic in a non-MPLS setting, where you make the IGP metric look better on the side with QoS for traffic headed to a PBX subnet (ATM in one case). More recently, I’m looking at settings with IPT using Microsoft Office Communicator and soft-phones, or soft IPVC clients. In that case, I lean more towards dealing with such device by somehow marking at the edge or trusting DSCP markings. And then you can do DSCP-based PBR on the site legacy router, to “deflect” such traffic to the MPLS router. If you track objects (MPLS neighbor and perhaps a route), you can not deflect traffic via PBR if the MPLS side is down. (Admittedly, a topic worthy of more discussion in and of itself.)

If you have two routers at the site, things get more complex. If you only want selected traffic using the MPLS side, then don’t redistribute the EBGP into the site IGP on the MPLS router. Do still run the IGP on the MPLS router. If you have default routing advertised on the legacy side, then traffic (one way or another) will go to the legacy router, and you can use PBR as above to “deflect” certain traffic to the MPLS router. If the router doesn’t have a route to the destination but the legacy router does, the MPLS router will send it back. You can avoid a routing loop if the preferred path back is different than the one you’re doing PBR on. A dedicated router to router link rather than one via the site-interface is one way to do that. You may have to use point-to-point interfaces to the L3 site switch(es), rather than a VLAN, to avoid having the two routers peer with each other over a VLAN they also communicate with the rest of the site or site switch(es) over. Tracking the MPLS router can disable the PBR if the MPLS router goes down.

Note: Loss of default over the legacy WAN will mess this up. You could have the legacy WAN router advertise default for the site, being careful to NOT advertise default into the legacy WAN. An alternative is to have the MPLS router redistribute prefixes into the IGP. Since they’re external, the legacy router will be preferred unless a prefix stops being advertised on the legacy side. In that case, all traffic for that site will start being routed to the MPLS side. You might want an access list to prevent data traffic from using MPLS, Or a QoS policy that protects your IPT / IPVC, and gives remaining bandwidth for limited data use.

Sites Becoming Transit

We haven’t really talked about this yet, but sites tend to redistribute the site IGP into EBGP to get the site prefixes to be known across the MPLS WAN. The alternative is to configure a static site summary route and carefully redistribute that and only that static route into the EBGP.

The site IGP redistribution approach amounts to bi-directional redistribution. This can be a receipe for routing loops, and is generally best done with some distribute lists and filtering. The problem becomes, configuring an access list or prefix list to only advertise out the site prefixes is messy, making the static summary redistribution above look simpler. In general, I don’t like having to do site-specific filters, it creates extra operational work and can be a source of problems.

When you have two WAN connections, conceivably an outage might lead to a remote site (probably with limited access bandwidth) becoming transit, as in traffic going via one WAN to the site to get to a site on the other WAN. Routing metrics generally should take care of this. If you’re going IGP into MPLS redistribution with too attractive a metric, the routing metrics may not work in your favor. This is the common problem when doing bi-directional redistribution.

One form of insurance against this is to tag routes being learned at a site or redistributed into the IGP. You can then use the EIGRP or OSPF tags to prevent such remote routes from being advertised back into the WAN, so that only the site prefixes are advertised. This does take consistent configuration. The good news is that the route maps and distribute lists needed can be made site-neutral, i.e. part of a cookie-cutter configuration template.

Coming Topics

A lot of the above discussion carries over the the other scenarios above, with some note-worthy differences in how it affects you. We’ll save specific discussion for the follow-on blog(s) about this.

References

Ivan Pepelnjak has a great site with lots of information, and I have huge respect for the depth of his knowledge of routing. He’s got a number of good and relevant articles, see for instance http://blog.ioshints.info/2008/12/basic-mpls-vpn-design-configuration.html. I come at this a little differently, with less focus on the MPLS provider side, and more on simplicity for the customer side (to the extent possible). I like the validation of seeing some of his discussions of customer side factors (MPLS security, VPLS service, etc.) — although I slightly disagree with the emphasis of the MPLS security article. MPLS Security takes a little more work by the provider than FR or ATM perhaps — but should be attainable. (I worry about sites unnecessarily doing IPsec with all its burdens, somewhat defeating the value of high speed MPLS service.) I love the title “VPLS is not Aspirin”.

Cisco has a good posting about buying MPLS services, the Enterprise Consumer Guide, at http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/L3VPNCon.html. It’s a good source of questions to ask vendors about in an RFP, although I recently thought of a number of things I wanted to ask that might not have been in there. (And there was an amazing first in dealing with WAN provider discussions: we actually got some darn good answers, and I hereby award the prize for best technical answers in a customer document to Level 3.) There is a longer (more detailed?) version available as a book from Cisco Press. See http://www.amazon.com/Selecting-MPLS-Services-Chris-Lewis/dp/1587051915.