Villa A near St. Jean
Let's start with a villa near St. Jean. The ADSL line had a raw upstream speed of 1 Mbit/s and raw downstream speed of 15.6 Mbit/s, using the latest ADSL2+ technology. These speeds are pretty good and tell that the other end of the ADSL line is less than 2 km away. The line uses PPPoA technology which means a lot of layers, your IP packet will be inside a PPP frames, inside ATM frames and inside the ADSL frames.
The PPPoA technology gives the option to terminate the customer connection further away, if it made any sense, the next hop of the connection from the villa in St. Barth could be in New York. Sometimes it makes sense to terminate the connection further away to make administration easier.
So how does the traffic flow from Villa A?
Let's start with a traceroute to a finnish newspaper web site:
traceroute to
www.hs.fi (158.127.30.40), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 2.178 ms 0.991 ms 1.336 ms
2 net1lo-bidon.bsami113.amiens.francetelecom.net (193.253.160.3) 24.693 ms 25.605 ms 24.819 ms
3 80.10.245.253 (80.10.245.253) 25.997 ms 26.746 ms 25.539 ms
4 pos0-3-0-0-100.nyktr3.newyork.opentransit.net (193.251.248.2) 75.289 ms 76.319 ms 74.629 ms
5 tengige0-3-0-6.nyktr1.newyork.opentransit.net (193.251.151.240) 72.888 ms 71.446 ms 72.857 ms
6 te0-0-0-12.ccr21.jfk07.atlas.cogentco.com (154.54.11.229) 70.540 ms 70.671 ms 71.915 ms
7 be2059.mpd22.jfk02.atlas.cogentco.com (154.54.1.221) 71.038 ms
be2056.ccr21.jfk02.atlas.cogentco.com (154.54.44.217) 71.133 ms
be2057.ccr22.jfk02.atlas.cogentco.com (154.54.80.177) 70.623 ms
8 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 76.504 ms 76.425 ms
be2097.ccr22.bos01.atlas.cogentco.com (154.54.30.118) 76.849 ms
9 be2386.ccr21.lpl01.atlas.cogentco.com (154.54.44.162) 161.214 ms
be2387.ccr22.lpl01.atlas.cogentco.com (154.54.44.166) 155.340 ms 155.650 ms
10 be2182.ccr41.ams03.atlas.cogentco.com (154.54.77.245) 158.531 ms
be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70) 156.333 ms 155.327 ms
11 be2187.ccr42.ham01.atlas.cogentco.com (154.54.74.126) 163.529 ms
be2186.ccr41.ham01.atlas.cogentco.com (154.54.74.122) 160.751 ms
be2187.ccr42.ham01.atlas.cogentco.com (154.54.74.126) 164.247 ms
12 be2281.ccr21.sto03.atlas.cogentco.com (154.54.63.2) 184.862 ms 184.080 ms
be2282.ccr22.sto03.atlas.cogentco.com (154.54.72.106) 182.787 ms
13 be2396.ccr21.sto01.atlas.cogentco.com (130.117.50.122) 184.255 ms 180.472 ms 181.148 ms
14 cgnt-130.kn1.sth.portlane.net (149.6.168.130) 176.175 ms 176.720 ms 179.443 ms
15 rai1-tr1.dnaip.fi (62.78.107.175) 194.278 ms 199.186 ms 194.742 ms
16 hel5-tr3.dnaip.fi (62.78.107.12) 182.745 ms 182.071 ms 186.221 ms
17 hel1-tr2.dnaip.fi (62.78.107.124) 191.398 ms 191.039 ms 187.591 ms
18 * * *
Traceroute is one of the basic tools to see how traffic flows at the IP level. The time for each hop tells the round trip time, how many ms it takes for a packet to reach that hop and come back. As there's a limit how fast light goes, the rtt can give an idea how far we've gone, for example when we cross the ocean.
Hop #2 tells that Villa A has ADSL from Orange (France Telecom). This also tells that the PPPoA layer is not terminated in St. Barth but all traffic from the Villa goes to St. Maarten before it enters France Telecom's IP network. Easier for administration.
Hop #3 is the router to France Telecom's OpenTransmit global backbone and on Hop #4 and #5 we're already in New York! There's no direct link between St. Maarten and New York but again the IP network and physical network are separated, just like there's a physical link between St. Barth and St. Maarten. Thehe IP traffic from the villa goes straight to St. Maarten.
The physical connection from St. Maarten to New York is most likely using the GCN, Global Carribean Network, that connects a lot of Caribbean islands with 2.1 Tbit/s capacity and further to the US from Puerto Rico with other links. Puerto Rico is a well connected hub in the region. For example the new Pacific Caribbean Cable System, PCCS, with 80 Tbit/s of capacity is there.
From New York it's business as usual, just like any other France Telecom customer. France Telecom uses Cogent's network to get to North Europe; JFK to Boston, from Boston to Liverpool, further to Amsterdam, to Hamburg and Stockholm. That's not very straightforward but hey.. In Stockholm the finnish newspaper's carrier (DNA) connects to Cogent's network. You can see the hop across the Atlantic at #9 when the round trip time increases about 90 ms.
Let's take something more interesting. How about traffic to Google from the Villa A?
traceroute to
www.google.com (173.194.67.105), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 1.536 ms 1.030 ms 1.031 ms
2 net1lo-bidon.bsami113.amiens.francetelecom.net (193.253.160.3) 25.347 ms 25.587 ms 25.203 ms
3 80.10.245.253 (80.10.245.253) 24.935 ms 25.139 ms 25.159 ms
4 pos0-3-2-0-100.nyktr3.newyork.opentransit.net (193.251.247.233) 75.819 ms 75.390 ms 72.142 ms
5 tengige0-3-0-6.nyktr1.newyork.opentransit.net (193.251.151.240) 73.609 ms 73.659 ms 75.742 ms
6 tengige0-7-0-12.madtr1.madrid.opentransit.net (193.251.241.193) 172.982 ms 180.671 ms 171.615 ms
7 72.14.202.58 (72.14.202.58) 169.547 ms 169.236 ms 170.399 ms
8 209.85.252.150 (209.85.252.150) 171.562 ms
209.85.251.242 (209.85.251.242) 169.503 ms 169.680 ms
9 209.85.240.191 (209.85.240.191) 160.090 ms 160.534 ms 160.339 ms
10 209.85.248.200 (209.85.248.200) 169.613 ms
209.85.245.87 (209.85.245.87) 172.167 ms
209.85.254.62 (209.85.254.62) 171.715 ms
11 209.85.250.163 (209.85.250.163) 168.157 ms
wi-in-f105.1e100.net (173.194.67.105) 170.276 ms 171.056 ms
Again all the traffic hits St. Maarten right away and to New York. From New York we continue in France Telecom's OpenTransit network to Madrid across the Atlantic (again notice the increased rtt). In Madrid we hit a number of unnamed 209.85.x.x hops that belong to Google. 1e100 (hop #11) is Google's network (just google what google means).
What exactly happened? Google from St. Barth goes to Spain? Yes it does. When your browser asks for the IP address to Google, they see that the request comes from France Telecom's network and give you Google address that would be close to France. So when you type
www.google.com on Orange ADSL connection in St Barth to your browser, you'll go through Spain to Google's network.
Again France Telecom hides the IP network and physical network quite well, there isn't a direct cable between New York and Madrid.
What about Facebook then, does it go to Spain as well? Let's see (
www.facebook.com)
traceroute to star.c10r.facebook.com (31.13.69.176), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 1.386 ms 1.066 ms 1.001 ms
2 * net1lo-bidon.bsami113.amiens.francetelecom.net (193.253.160.3) 26.342 ms *
3 80.10.245.253 (80.10.245.253) 25.499 ms 25.549 ms 24.824 ms
4 pos0-3-0-0-100.nyktr3.newyork.opentransit.net (193.251.248.2) 74.148 ms 74.197 ms 71.921 ms
5 tengige0-3-0-6.nyktr1.newyork.opentransit.net (193.251.151.240) 72.157 ms 72.320 ms 71.157 ms
6 ae10.edge2.newyork.level3.net (4.68.70.169) 70.285 ms 70.239 ms 69.645 ms
7 vlan80.csw3.newyork1.level3.net (4.69.155.190) 77.200 ms 76.340 ms
vlan70.csw2.newyork1.level3.net (4.69.155.126) 76.824 ms
8 ae-81-81.ebr1.newyork1.level3.net (4.69.134.73) 78.320 ms
ae-91-91.ebr1.newyork1.level3.net (4.69.134.77) 76.992 ms 76.097 ms
9 ae-4-4.ebr1.newyork2.level3.net (4.69.141.18) 78.771 ms
ae-56-56.ebr2.washington12.level3.net (4.69.201.62) 77.359 ms
ae-46-46.ebr1.newyork2.level3.net (4.69.201.42) 76.528 ms
10 ae-40-40.ebr2.washington1.level3.net (4.69.201.93) 78.146 ms
ae-37-37.ebr2.washington1.level3.net (4.69.132.89) 76.507 ms
ae-40-40.ebr2.washington1.level3.net (4.69.201.93) 77.620 ms
11 ae-72-72.csw2.washington1.level3.net (4.69.134.150) 77.424 ms
ae-92-92.csw4.washington1.level3.net (4.69.134.158) 77.637 ms
ae-72-72.csw2.washington1.level3.net (4.69.134.150) 78.317 ms
12 ae-4-90.edge2.washington4.level3.net (4.69.149.208) 77.309 ms
ae-3-80.edge2.washington4.level3.net (4.69.149.144) 78.612 ms
ae-2-70.edge2.washington4.level3.net (4.69.149.80) 77.438 ms
13 facebook-in.edge2.washington4.level3.net (4.53.114.46) 77.357 ms 76.071 ms 76.814 ms
14 ae2.bb01.iad1.tfbnw.net (204.15.20.120) 76.515 ms
ae1.bb02.iad1.tfbnw.net (74.119.79.204) 77.584 ms
ae3.bb02.iad1.tfbnw.net (74.119.78.60) 78.521 ms
15 ae1.dr03.ash3.tfbnw.net (204.15.21.106) 77.716 ms 79.981 ms 76.711 ms
16 po126.msw01.08.iad1.tfbnw.net (31.13.29.155) 90.485 ms 97.445 ms 77.303 ms
17 edge-star-shv-08-iad1.facebook.com (31.13.69.176) 77.032 ms 76.453 ms 76.557 ms
Again we head to New York in France Telecom's network but the traffic doesn't go to Europe but to Level3's network right there in New York.
In Level3's network (hops #6 to #13) we travel through New York to Washington DC, where Facebook is connected to Level3's network. TFBNW? The FaceBook NetWork. The "ash" and "iad" in the naming indicate that it's the rental facility Facebook has in Ashburn, VA, from DuPont Fabros Technology. You'll find Yahoo and MySpace at the same location as well. DFT actually runs at least six data centers there, ACC2 to ACC7.
We see the round trip time to hit about 70-80 ms when we go from St. Maarten to New York but it never really increases above that. We never go across the Atlantic.
One more example, just to give a picture how network connect to each other
traceroute to
www.sonera.fi (194.251.244.241), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 1.633 ms 1.366 ms 1.317 ms
2 net1lo-bidon.bsami113.amiens.francetelecom.net (193.253.160.3) 25.537 ms 24.806 ms 25.499 ms
3 80.10.245.253 (80.10.245.253) 25.048 ms 25.474 ms 25.015 ms
4 pos0-3-2-0-100.nyktr3.newyork.opentransit.net (193.251.247.233) 73.413 ms 71.841 ms 71.107 ms
5 tengige0-3-0-6.nyktr1.newyork.opentransit.net (193.251.151.240) 70.931 ms 71.879 ms 71.614 ms
6 nyk-b5-link.telia.net (213.248.103.1) 70.794 ms 71.041 ms 71.843 ms
7 nyk-bb1-link.telia.net (213.155.135.18) 115.617 ms 70.806 ms 70.876 ms
8 kbn-bb3-link.telia.net (80.91.247.116) 173.021 ms 176.035 ms 176.330 ms
9 s-bb3-link.telia.net (213.155.136.173) 180.741 ms
s-bb3-link.telia.net (213.248.65.141) 186.364 ms
s-bb3-link.telia.net (62.115.136.187) 183.061 ms
10 hls-b1-link.telia.net (213.248.64.102) 189.066 ms
hls-b1-link.telia.net (80.91.247.219) 190.591 ms
hls-b1-link.telia.net (213.155.133.73) 190.158 ms
11 tsf-ic-155157-hls-b1.c.telia.net (213.248.68.210) 194.937 ms 198.770 ms 198.351 ms
12 141.208.25.30 (141.208.25.30) 181.533 ms 181.917 ms 182.167 ms
13 finn.sonera.eu (194.251.244.241) 197.131 ms 193.603 ms 194.817 ms
The Finnish part of TeliaSonera carrier. Again all the way to New York but right in New York France Telecom connects to TeliaSonera's global network. In TeliaSonera's network we head to Copenhagen, Stockholm, and eventually Helsinki. The finnish carrier goes pretty straightforward to Northern Europe but the finnish newspaper traffic travelled all around Europe. That's the Internet.
All the big players with global networks talk to each other in different places around the globe, exchange traffic when it makes sense, and sometimes use other's network to reach the destination.
As we can see, the round trip times to everywhere are pretty normal. There isn't a single connection in the network infrastructure that is saturated with too much traffic. Using the consumer oriented SpeedTest from Ookla we can see that from the Villa A we get about 12-13 Mbit/s download speed and .9 Mbit/s upload speed from all the way from New York.
With all the layers mentioned earlier this is pretty much as fast as one can go on the ADSL line to the villa. The layers add overhead but the performance is the same to St. Maarten as it's to New York. That's enough bandwidth to watch Netflix in HD.
In summa summarum, Villa A had pretty much the same Internet access as one would get in a suburb somewhere in the US or Europe.