Tuesday, November 23, 2010

How do they deliver files so fast?

TomTom emailed me this morning to say a new map was available for my GPS Receiver. Since we're just about to go on a trip, I jumped at the chance for a shiny new map. I use a TomTom Go 720.

1. The download rate was impressive. My TimeWarner Business Class service in the Raleigh NC area provides 8 Mbps download capacity. At 8:22 am, this transfer was running at 8 Mbps. That suggests the bottleneck was really my access link.

2. The time-to-live on received packets was 57. That's a fairly close server, at only 8 hops away.

# 2010-11-23 08:25:50 mark@nabla ~$ ping -m 7
PING ( 56 data bytes
36 bytes from ae-1-0.pr0.dca20.tbone.rr.com ( Time to live exceeded
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 40 5400 8fbd 0 0000 01 01 0464

Request timeout for icmp_seq 0
36 bytes from ae-1-0.pr0.dca20.tbone.rr.com ( Time to live exceeded
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 40 5400 afa8 0 0000 01 01 e478

--- ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

# 2010-11-23 08:25:55 mark@nabla ~$ ping -m 8
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=57 time=271.677 ms
64 bytes from icmp_seq=1 ttl=57 time=281.295 ms
--- ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 271.677/276.486/281.295/4.809 ms

3. TimeWarner / RoadRunner does such a nice job naming their point-to-point router IPs, we can guess about the network technology used.

traceroute to (, 64 hops max, 52 byte packets
1 ( 0.813 ms 0.595 ms 0.586 ms
2 ( 231.388 ms 268.237 ms 268.356 ms
3 gig9-1-4.rlghncg-rtr1.nc.rr.com ( 260.497 ms 279.656 ms 253.422 ms
4 pos1-0.rlghnca-rtr1.nc.rr.com ( 254.433 ms 258.332 ms 264.331 ms
5 ge-6-0-0.rlghncpop-rtr1.southeast.rr.com ( 257.150 ms 274.931 ms 265.894 ms
6 ae-3-0.cr0.dca10.tbone.rr.com ( 259.336 ms 317.687 ms 286.754 ms
7 ae-1-0.pr0.dca20.tbone.rr.com ( 290.598 ms 274.169 ms 280.447 ms
8 209-18-42-26.dca20.tbone.rr.com ( 287.974 ms 254.063 ms 230.003 ms

They use the term "ae", which is a Juniper term for "Aggregate Ethernet". So I'd guess TimeWarner is using Juniper routers, at least in the "dca" area. (Washington DC?)

The third-hop interface is GigabitEhternet 9/1/4 on Raleigh NC "G" Router 1.

The fourth-hop interface is Packet-over-Sonet 1/0 on Raleigh NC "A" Router 4.

The fifth-hop interface is Gigabit Ethernet 6/0/0 on Raleigh NC "POP" router 1.

The sixth-hop interface is AggregateEthernet 3/0 on "cr0" in "dca10". I'm guessing that's "core router 0" in Washington DC area 10, but I really don't know. The router naming scheme changed when we left the Raleigh NC area.

The seventh-hop interface is Aggregate Ethernet 1/0 on "pr0" in "dca20". That sounds like a peering router, and it'd make sense, since that's the router that is apparently connected to the file server.

4. The bulk of the data came from came from, aka "209-18-42-26.dca20.tbone.rr.com.".

According to ARIN, somebody named that network "COUDERSPORTBB-1". Coudersport PA was the location of Telcove and Adelphia Communications, and currently an Operations Center for Level(3).

5. The Ethernet frames, as they arrive, are 1514 bytes. They have 1488 bytes of TCP payload in each packet. That's making full use of 1514 byte frames, and it's as efficient as they could be without special tricks.

6. The IP datagrams sent have the "Don't Fragment" bit set. That suggests they used Path MTU discovery to optimize the right packet size.

What's the point? The file server sending me this file is directly connected to the RoadRunner / TimeWarner network. And it's likely that at this hour, the TimeWarner network is not saturated.

No comments: