the shrinking internet

Is the Internet shrinking? When the net was first developed, it was tiny. Just a few interconnected servers made up the entire system. Then, as the web created new interest in the global public network, Internet service providers and online content exploded. The web ballooned in size. However, what's interesting is not so much how big it is, as how many organizations control the traffic.

The Internet is a network of networks. It is comprised of lots of smaller networks that contain detailed information about how to route information internally. These networks, known as autonomous systems, then connect together to route information between each other, creating the broader public Internet.

The number of these networks, as represented by individual autonomous system numbers (ASNs), has grown substantially. There were less than 5000 of these networks in 1999. Now, there are almost 40,000 ASNs.

However, a study by the Internet Engineering Task Force (IETF) and Arbor Networks last year revealed that the number of ASNs controlling the majority of the traffic has fallen. In 2007, thousands of ASNs contributed 50% of the content. However, two years later, 150 ASNs contributed 50% of all Internet traffic, the study revealed.

Two factors contribute to this consolidation of traffic. The first is the tendency of things to go viral. Sites become popular because as they gain users, those users draw others in. This is how social networking operates. Companies such as Facebook reach a certain level of popularity and then begin growing by pure momentum.

The second factor feed on the first, and has to do with the aggressive nature of acquisition on the Internet. Large companies tend to buy other companies based on their rapidly growing traffic. Google invested hundreds of millions in YouTube for this reason. The company may not have made money from the video site for years after the acquisition, but it was investing in its ability to draw traffic–and its potential to become a viral medium.

All of this has led to a new type of Internet, that had developed in the last few years, featuring a number of relatively young players, such as Google, Facebook, Yahoo, Microsoft, and content distribution networks. These companies all own their own automated systems - the networks that connect together and make up the larger Internet. So, what part do they play in this new landscape?

Originally, local Internet providers would exchange traffic with each other via regional Internet providers who also served them with bandwidth. Those regional providers would in turn exchange information via network access points, which were central transit points for Internet data. Those network access points would send information via the national backbones operated by players such as Sprint and MCI.

As these larger players have emerged, each owning their own automated systems, the situation has changed. In many cases, the huge amounts of traffic flowing to these ‘hyper giants’ is still exchanged via the modern version of network access points, which are privately-owned Internet exchange points. However, these large players are exchanging an increasing amount of this traffic directly with each other, and with tier two or three internet service providers.

As this happens, the traditional hierarchy between user-facing websites and back-end exchange networks is collapsing. The sites that serve traffic directly to Internet users are also becoming those that exchange at the backend. Power is consolidating. Facebook, one of the most consumer-facing sites of all, represented a quarter of all US Internet page views in November last year. These, and content distribution networks designed to make the distribution of Internet-hosted data more efficient, are becoming the new backbones of the Internet.

What does this mean for the egalitarian nature of the Internet? It was originally intended to be a distributed system that routed information around damage. Now, however, an increasing amount of information is being hosted by and exchanged between a relatively small number of very large, consumer-focused companies. Google is among them. In 2008, just under a third of its traffic was exchanged via transit networks such as Internet exchange points. However, last year, this number had risen to 60%.

This power law distribution is perhaps a natural development in a market-driven environment such as the Internet. But what does it mean for the security and reliability of the system? Aggregation of traffic also means a smaller number of failure points. Are we losing the original resilience of the Internet? And if so, what can be done about it?

Stewart,

Photo: The opte project