Making WAN-wide VLAN-style VPN's in an L3 world

These things are called VPN's, and they do live up to the name in terms of how they appear to the customer, but they're not very much like IPsec VPN's because there's no encryption. One has to trust the integrity of the carrier's network, which probably passes through competitors' basements and AT&T-run NSA monitoring rooms. Anyway, encrypted or not, everyone seems to want them.

How could an ISP sell the same thing without STP? Well, if the customer will cooperate a bit, and if the ISP's switches support it, they could use what Cisco calls VRF instances. (My Extreme switches don't support it.) This scheme gathers a bunch of interfaces on a switch into a named VRF group. IP addresses can be reused between VRF groups. Packets can't cross VRF boundaries. Each VRF instance will run its own OSPF process. This is less convenient than VLAN's in several ways:

• STP extensions like MST and EMISTP allow one to share the routing decisions a single STP instance makes about blocking and unblocking ports across hundreds of VLAN's---this allows convergence with less CPU time and fewer STP packets. There isn't an analagous trick for saving the CPU time and HELLO traffic needed by so many separate OSPF processes. I think ten VRF's would work, but hundreds of VRF's each running their own OSPF process probably wouldn't work well.
• The ISP can no longer sell customers something as convenient as an apparent L2 VLAN. It'll have to configure IP's onto its own switches that match the customer's IP schemes, leading to endless fist-wringing conference calls with customer imbeciles. The ISP will even have to participate in the customer's IGP, which is a disaster unless they somehow turn this into a selling point, ``We'll manage your WAN for you, for more money.''
• Adding a VLAN to the old large STP domain as an easy, quick task that takes minutes. Adding a new VRF is a long, complicated, mistake-prone task that takes hours and might need a maintenance window.

  For example, the fibers between switches will be L2 trunks with one tagged VLAN per VRF instance. These VLAN's will exist nowhere except on these trunks. Just assigning VLAN numbers to these trunks is a nightmare---will you run out of VLAN's? 50 trunks, 100 customers, and you're out of VLAN numbers! Since all these trunk VLAN's contain only two ports and span only two switches, you can reuse VLAN numbers a bit if you want, but then you have a graph-coloring problem. And this is just for picking the number you want to assign to the VLAN you're adding to the trunk---you haven't even assigned IP's to each end of the trunk yet (that'll take a conference call with the customer!)

Clearly this isn't good enough for a MAN that wants to have hundreds of customers. So, they could use MPLS instead. I don't know much about that yet.

From: Andy Smith
Subject: L2TPv3
To: Miles Nordin
Date: Fri, 14 Mar 2008 16:06:25 -0400
X-Mailer: Apple Mail (2.919.2)

At http://web.ivy.net/~carton/rant/l3-switch/ you wrote:

> It's possible, and I believe a best-practice, to avoid large STP  
> domains. Really it's best to avoid STP period. L3 switches make this  
> possible, but the obvious configuration gives up the VPN features  
> these big organizations became accustomed to getting from their  
> oversized STP domains.

Well, first, it's good to see you've entered the 21st Century and have  
condoned L3 ethernet switching.  :)

You are absolutely correct that STP is something to be avoided,  
particularly on large Metro-E SP networks or even big corporate /  
enterprise LAN's.  But you fail to even mention the simple alternative  
- L2TPv3.  With L2TPv3 a pseudowire is as easy to create as any other  
sort of tunnel.  It's the best part of MPLS (L2 VPN's) with the worst  
parts of MPLS (TE and tag switching) left out.

-Andy