You didn’t have to be a customer of AT&T Internet to find that an ATT Internet outage disrupted your Internet communications this morning.
AT&T, which also includes Pacbell (Pacific Bell) and SBC Global, accounts for a significant portion of Internet routing and traffic between Internet sites in the San Francisco Bay area, and so when parts of the ATT Internet infrastructure go down, traffic between all sorts of places on the Internet gets disrupted and falls into a black hole.
For nearly 2 1/2 hours this morning, AT&T Internet routing stopped working. For those who are familiar with tracerouting, a typical traceroute between sites needing ATT to navigate the Internet between point A and point B looked like this:
1 192.168.1.1 (192.168.1.1) 2.944 ms 2.062 ms 1.760 ms
2 adsl-[private IP address].dsl.snfc21.pacbell.net ([private IP address]) 11.376 ms 10.127 ms 10.583 ms
3 dist2-vlan50.snfc21.pbi.net (206.171.134.131) 11.243 ms 10.500 ms 9.940 ms
4 bb2-10g2-0.snfcca.sbcglobal.net (216.102.176.226) 10.137 ms 10.457 ms 9.915 ms
5 bb1-p4-0.snfcca.sbcglobal.net (151.164.190.189) 10.932 ms 10.531 ms 10.661 ms
6 core1-p8-0.crsfca.sbcglobal.net (151.164.92.181) 10.425 ms 11.159 ms 10.006 ms
7 bb1-p8-0.crsfca.sbcglobal.net (151.164.243.2) 10.066 ms 10.895 ms 10.448 ms
8 ex1-p3-0.eqsjca.sbcglobal.net (151.164.41.101) 133.837 ms 206.346 ms 215.055 ms
9 * * *
10 * * *
11 * * *
Those *s are the Internet routing equivalent of “you can’t get there from here”, or, perhaps, “the bridge across that river has washed out.”
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
While nobody has yet identified exactly what happened – or taken responsibility – things started going wrong with AT&T routing just before 6:00 a.m. Pacific time. The outage remained until about 8:15 a.m.
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
Just a side-note on the ‘*’ in traceroute output. It doesn’t necessarily mean that there is nothing there, it can mean that a router (or a string of them) in the path are either configured intentionally to not respond to traceroute’s probe packets or are responding in a fashion that some other machine in the return path is seeing as bogus. An example of the latter would be the use of private addresses in routers in a way that causes their ICMP responses to be sent with source addresses in a RFC1918 network that some other routers and firewalls reasonably consider bogus and drop-worthy.
In the end, it is not definitive that packets are traveling off into nowhere just because traceroute trails off into asterisks.