LINX, The London Internet Exchange, is the UK's largest Internet exchange point. Last night LINX suffered a partial outage, causing Internet traffic jams that affected many UK ISPs.
On a normal day LINX handles 500 Gbps of traffic, a large proportion of the UK's Internet traffic load. LINX is a non-profit company run for the benefit of its members - ISPs, hosters and carriers - who need to exchange Internet traffic between each other.
With this important exchange having some technical issues, how were we able to react to minimise the impact on our customers? Firstly as LINX members Xtraordinary receives frequent status reports from LINX technical staff and we also participate in a private mailing list for LINX members. This first hand information allowed us to take an informed decision on appropriate technical measures to take.
LINX operates two independent networks - called the Brocade and Extreme rings. These two rings each traverse ten datacentres across Docklands, central London and out through west London as far as Slough. Each ring runs on a different brand of ethernet switch equipment. This provides not only physical independence but vendor and software independence. This reflects LINX's robust design - if there were only one ring and one vendor of switch equipment then the entire exchange might be susceptible to software or hardware flaws affecting that particular brand of equipment.
Xtraordinary is connected to both LINX rings, each at 1Gbps, connecting in our London hosting site at Interxion London City Datacentre near Bishopsgate.
Last night at 6.02pm we decided to turn off our port on the troublesome Brocade ring.
So what happens to all the traffic when we turn one of our LINX ports off? Traffic immediately redirects:
- to the 2nd LINX Extreme port where possible. Not all ISPs/hosters maintain two LINX ports, so this isn't an option for all of the traffic.
- to our upstreams - Level(3), TInet, Abovenet and Global Crossing - all played a part in taking up the slack. The larger carrier networks maintain private fiber inter-connects between each other that don't depend on LINX. Traffic always has the option to traverse between the carrier networks because we are paying for its throughput to the end destination.
- to other exchanges. If necessary our upstream networks can route traffic via other Internet exchanges in cities such as Amsterdam, Paris and Frankfurt. This doesn't appear to have been necessary yesterday.
As we have plenty of free capacity on our LINX and upstream connections turning off a port is not an issue for us.
This morning we are awaiting the all-clear from LINX before re-enabling our Brocade port.
Ivan Groenewald
CTO
