Storms in large STP domains</h3> </head> <body> <h3>Storms in large STP domains</h3> In neither case did they find the root cause of the problem, which in my opinion makes it even worse than Xerox's story. I like how Boston was operating their network, knowing it was looping near the edges, for months! In New Zeland, I like the image of techs running to AT&T House on Murphy St. to ``start unplugging things.''<p> Look, clearly the New Zeland guys are vastly more competent compared to the ``rally the team together'' no-sleep pager crap you get from the Boston crew. They knew what was going on from the start, systematically isolated the problem an order of magnitude more quickly than the Boston guys, fixed the important part of the network first, did their final resolution work with a network that was up not down, and did this all with the equipment they had rather than dumping in buckets of money and begging for TAC support. I don't share the reflexive respect some people seem to have for incompetent people allowing a crisis to develop, then working themselves and their underlings into an exhausted and confused lather---I know a lot of organizations where that douche's career would be advanced by this disaster because he ``handled the crisis well'' or ``showed dedication'' or some crap. <A HREF="http://www.flownet.com/gat/quotes.html">Furious activity is no substitute for understanding.</A><p> The NZ MAN guys also acknowledge subtle inconvenient things, like apologizing the crappy network was still good enough to keep BGP sessions up when it would have actually been better if it had failed completely, and, my favorite part, how the in-band management completely crumbled. The Boston guys seem to have no ability to infer the problems their users are experiencing except when their users know enough to complain. Most of the time, users don't know when to complain, or what to complain about.<p> so, yes, people still have big L2 domains and still have big problems with them.<p> But with modern equipment, the big-L2-domain STP loop problem is mostly fixed. Even modern L3 switches do still have L2 domains of tens of ports. I think the silicon would work fine with IPv4 /30-subnets-per-port, making the switch into more like an ISO CLNP IS-IS Level 1 router, but I haven't seen toolkits that make this manageable, so small, localized STP loops are still possible.<p> The edge configuration problems leading to STP screwups are actually worse. RSTP makes the proliferation-of-knobs problem better, but MST is stunningly complicated, and I've heard at least one story from a friend of a new hire's attempt to optimize some VLAN-to-STP assignment (not sure it was MST) causing a loop. Even with RSTP, we're still asked to manually mark so-called ``edge'' ports---it's just maybe a little smarter if you mismark something.<p> The unmanaged 3-port miniswitches inside VoIP phones that I predicted would become problems are ubiquitous now. These hide link state from the switch running STP so it doesn't know when people move cables around, so loops can still happen in practice at the edge (at least for a few seconds).<p> They've happened to me, on my very small LAN, but to be fair I'm using seven-year-old L3 switches. I have to disable STP (ExtremeWare 4.x switches don't do RSTP-style edge ports---you have to switch off STP entirely to avoid the forwarding delay) on some ports to avoid boot-from-network timeouts, and I confused one of these ports with a trunk to another switch. And I have VLAN's spanning multiple switches because I use IPv6, and Extreme's silicon can't L3-route that, even in the stuff they're selling in 2007 (it L3 routes IPv6 in software). so, if it weren't for those two things together, I'd have avoided the storm, or confined it to a single switch.<p> However, these old campus-wide outage problems are gone with L3 switches. There aren't weird ghost flapping loops caused by STP domains with too many hops. One can make VLAN's small---part of a floor, tens of ports---so the loop will only take down a single VLAN, or maybe an entire edge switch but still just one edge switch.<p> <hr> <ADDRESS><A HREF="./">L3 switches</A> / <A HREF="../../map.html">map</A> / <A HREF="../../">carton's page</A> / Miles Nordin <A HREF="mailto:carton@Ivy.NET"><carton@Ivy.NET></A></ADDRESS> <I>Last update (UTC timezone): $Id: switch-man-storm.html,v 1.1 2008/01/11 03:31:01 carton Exp $</I> </body> </html>