Network News

X My Profile
View More Activity

Major Internet Backbone Goes Down

Update, 5:53 p.m. ET: Cogent says the problem in the District was repaired at a little after 3 p.m. ET today and that the company expects the cut in its fiber line in New Orleans to be fixed within the hour.

Here's my post from earlier today:

Cogent, which runs one of the major Internet backbone networks that handles long-haul Web traffic, is currently down after its fiber network was somehow cut in two different locations. A look at Keynote's Internet Health Report site indicates serious problems across the board for Cogent. At the moment, even Cogent's homepage appears to be down from my location in Northern Virginia, serving up nothing more than a 404 File Not Found Error.

I spoke with Tecora Washington from Cogent. She confirmed that there were two fiber cuts; one that happened in New Orleans and a separate one in Washington, D.C. "This has effectively isolated the Southeast portion of Cogent's network."

Cogent spokesman Jeff Henricksen said in an e-mail that the network outage in New Orleans started this morning at 8:25 a.m. ET. He said work crews began repairs there at 1 p.m. ET today and that the company expects service to be restored within the next several hours. The Washington, D.C., outage began at 10:44 a.m. ET "after fiber there was damaged (crushed)." Henricksen said that damage should be repaired withing the next couple of hours.

This is creating some serious regional Internet traffic problems for some Internet service providers (ISPs), especially Comcast, according to individuals I spoke with today in Minnesota and in Pennsylvania. According to a thread over at Nanog, the outage is affecting ISPs from New York to Washington, D.C., and areas in the Southeast. The Internet Storm Center also is following this development. It is not clear when this problem will be fixed.

I'm not sure how often big fiber cuts like this happen at or near the same time on the same network at two major cities. Security Fix will update this post as more information becomes available.

What about you, Security Fix readers? Experiencing some serious slowness on your local network today? Drop us a line and let us know where you are and which ISP you use.

By Brian Krebs  |  November 17, 2005; 2:12 PM ET
Categories:  Latest Warnings  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Removing Sony's Software? Not Quite
Next: Researchers See Risk From Another Sony DRM

Comments

comcast is good here right now. 1pm eastern in North FL. yesterday i had an outage during the middle of the night.. it was time to go to bed anyway, but it was down for like an hour before i decided to give up.

Posted by: chris | November 17, 2005 12:51 PM | Report abuse

Seeing major issues between Level3 and Cogent ... could this be related to Level3 depeering Cogent in October and their supposed November 9th agreement?

Posted by: Level3 User | November 17, 2005 1:06 PM | Report abuse

From email just received:
"As of 11:30 this morning, both the splicing crew and the digging crew were
on site in New Orleans. The crews would not estimate the length of
time-to-repair at this time, but work has begun and the connection should be
restored today.

For the Washington DC cut, Cogent and MFN have identified the problem in an
OCF in 1110 Vermont and MFN has field technicians enroute to arrive at Noon,
EST. In the meantime, Cogent is looking to re-route traffic on Williams
fiber to restore connectivity while the crew repairs the cut."

Posted by: more details on Cogent cuts... | November 17, 2005 1:25 PM | Report abuse

Posted by: Lewvel3user-should read | November 17, 2005 1:51 PM | Report abuse

No problems with Comcast Broadband, Annapolis, MD. 6216 up/351 down. Tested on DSLreports.com, nac.net, New Jersey.

Posted by: Bill Hyland | November 17, 2005 1:55 PM | Report abuse

Comcast is slow in Seattle

Timeouts, and slow response, but most sites are coming up.

Posted by: Stuart MacMillan | November 17, 2005 2:07 PM | Report abuse

Slowdown, Silver Spring, MD. Comcast

Posted by: Guy W. | November 17, 2005 2:13 PM | Report abuse

Looks like Cogent's homepage is up, but interestingly nothing there about the outage. Comcast is okay in Columbia Heights area of NW Washington DC.

Posted by: Terry D | November 17, 2005 2:45 PM | Report abuse

Our T-1 in Alexandria, VA seems okay but our website (www.mortgagefool.com) and connecting email are still down (server in Herndon, VA)!

Posted by: Rob Maher | November 17, 2005 2:58 PM | Report abuse

I'm here in Los Angeles and our website and internet is still DOWN. Customer service told me DC cut line will be fixed in one hour but no ETA on New Orleans cut line

Posted by: Peter | November 17, 2005 3:04 PM | Report abuse

Here in Kansas City MO. Our connection seems to be OK but we can't see anything on the east coast. CNN, E-Trade, any Time Warner customers, etc....

We use Birch Telecom, the lease lines from SBC.

Posted by: Chris in KC | November 17, 2005 3:33 PM | Report abuse

As of our 3:30 EST, mortgagefool website & email (Herndon, VA) are back!!

Posted by: Rob Maher | November 17, 2005 3:36 PM | Report abuse

My ISP just went back up after being affected by this from the beginning--

Posted by: Rob Douglas - PrivacyToday.com | November 17, 2005 3:36 PM | Report abuse

FYI - All's well with Comcast Internet in Sarasota, FL.

If anything speed seems to be a bit better than usual today...

Posted by: Frank in Sarasota | November 17, 2005 3:44 PM | Report abuse

umm...does this kind of thing make anyone else a little confused about the fragility of the information superhighway which all business now circles around?

just asking.

Posted by: jaime | November 17, 2005 3:50 PM | Report abuse

Hi Brian,

I know that Cogent is saying their outage is regional, but we monitor their
home page and it's not available from any of our seven monitoring points,
which include views from NYC, San Jose, Phoenix, Pennsylvania, Northern
Virginia, London and Sydney. I posted a short item:

http://news.netcraft.com/archives/2005/11/17/fiber_cuts_cause_major_outage_a
t_cogent.html

Posted by: Rich Miller | November 17, 2005 4:23 PM | Report abuse

I can't wait until more critical infrastructure is placed on these fragile networks. With VOIP, an outage like this can not only take out general network access, like email, chat, web, but also voice communications (even 911 service) as well. Won't that be fun!

Posted by: what's next | November 17, 2005 4:28 PM | Report abuse

Today's outage has caused a great deal of pain for our clients. We made the choice to use Cogent because of its stability and cost-effectiveness. Today's events obviously have done some damage to the stability factor. We hope this is an isolated incident and share the concern that it seems awfully coincidental that two cuts would happen on the same day.

Posted by: Rich Forsen | November 17, 2005 4:32 PM | Report abuse

Interesting. It took almost 3 minutes to connect this morning, instead of the usual 10-15 seconds. Thanks for your column - I read it often.

Posted by: Comcast in Seattle | November 17, 2005 6:05 PM | Report abuse

Hi Brian,

I was reading the comments to your blog entry, and many people were
complaining about their sites being unavailable, while others commented
about the fragility of the internet.

That's not quite the case. The internet is designed to route around this,
and it did, at least for us. We place our servers in a colocation facility
and we made sure to choose a facility with multiple internet feeds. One of
their main feeds is Cogent, but they use others like Level3 and Yipes. I
just did a traceroute and usually when I traceroute to our servers, it
travels on the Cogent backbone. Today it's going over Yipes. We do about
1 million page-views a day and our customers are located around the world,
and not a single complaint has come in about our sites being
inaccessible.

The trick is to ensure that wherever you host your sites is a place with
more than one internet feed coming in, each from different providers,
coming from physically different directions, and on different pipes. Most
Tier-1 datacenters do this.

Posted by: Justin Fidler | November 17, 2005 6:34 PM | Report abuse

thank you very much for your help. You guys 31407 rock, thanks again.

Posted by: Tuki Medaber | September 21, 2006 12:42 AM | Report abuse

thank you very much for your help. You guys 31407 rock, thanks again.

Posted by: Tuki Medaber | September 21, 2006 12:51 AM | Report abuse

Service through Charter Communications cable internet service has been down for 2 days, and is slowly starting to come up now. This is the first time I've been able to access the 'net since the night before last.

Posted by: Weeks | September 26, 2006 4:58 PM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company