Network News

X My Profile
View More Activity
About this Blog   |   On Twitter   |   Follow us on Facebook   |   RSS Feeds RSS Feed

Northrop to pay $250,000 for study of August computer outage

Rosalind Helderman

Defense contractor Northrop Grumman has agreed to pay up to $250,000 to study why Virginia's state computer networks suffered a widespread outage at the end of August and it then took days to restore data to some state agencies, Chief Information Officer Sam Nixon told lawmakers during a briefing Monday morning.

Nixon told members of the Joint Legislative Audit and Review Commission that he was confident the figure would cover the costs of a full soup-to-nuts study of the computer outage.

An initial failure of a memory card in a data storage unit housed south of Richmond knocked some processes offline at 26 of 89 state agencies. The Department of Motor Vehicles had to turn away customers for almost a week.

Sam Abbate, a vice president of Northrop Grumman, which holds a $2.4 billion contract to run and modernize the state's computer network, told the delegates and senators that the company regrets the inconvenience to Virginians. "What should not have happened, happened," he said.

Sen. R. Edward Houck (D-Spotsylvania) suggested the company buy full page ads in state newspapers to apologize to residents--similar to ads the company took out in 2009 to defend itself against allegations of poor service. And Sen. Janet D. Howell (D-Fairfax) questioned how the failure could have happened when the state was promised multiple layers of backups to its new, modern system. "We were promised redundancy, and we're not getting it--and we're paying for it.," she said.

Nixon said much of the weeklong delay resulted from the complicated process of restoring and rebooting data corrupted during the initial failure. Computer experts had determined some data has been lost--notably 4,200 photos of customers who visited the Department of Motor Vehicles for new licenses on the afternoon of Aug. 25, when the memory card first failed. The DMV already has asked those customers to return and have their photos retaken.

Northrop Grumman meanwhile is paying a Minnesota company that ordinarily works with NASA in an effort to restore a much larger pool of corrupted DMV photos taken during the four days leading up to the outage, Nixon said Monday. He said that work is progressing well and he is hopeful that customers who visited the DMV during Aug. 21-24 ultimately will not need to have their pictures retaken.

"I'm cautiously optimist that won't be necessary," he said.

Lawmakers also discussed possible fines that could be imposed on Northrop Grumman as a result of the outage. Nixon said he believes the company can be assessed at least $100,000 for violating contract provisions that require that none of the state's 4,800 servers go down for more than four hours a month. The state also will likely be reimbursed some costs as agencies are refunded their payments for routine computer services that could not be performed by the company while computers were down.

Gov. Bob McDonnell (R)'s chief of staff also has asked agencies to compile a list of other costs incurred, including overtime payments to staff who worked around the clock to restore services. Nixon acknowledged, however, that Northrop can likely only be forced to pay for those costs if the company's negligence is found to have caused the outage. He said the company's contract is "vague" as to what kind of conduct would constitute negligence; the governor's office is in discussions with Attorney General Ken Cuccinelli on the point, he said.

The first indications of failure occurred when the data storage unit, a piece of hardware about the size of eight refrigerators, alerted its manufacturer, Massachusetts-based EMC Inc., that one of two memory cards in the device was experiencing trouble, he said.

A technician in the Chester facility, acting on instructions from EMC, then replaced the card. But as soon as the card was replaced, the memory devices failed altogether. Subsequent testing indicated problems with the memory card that had not been replaced, he said, raising the possibility that the wrong card was replaced or that both cards failed simultaneously.

By Rosalind Helderman  |  September 13, 2010; 3:00 PM ET
Categories:  Robert F. McDonnell , Rosalind Helderman  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: NRCC ad hits 'phony Perriello' on lobbyist donations
Next: Wilder launches blog with warning on federal spending


While this equipment supports hot maintenance, no one in their right mind replaces these critical components without taking the storage offline. Our tax dollars go to pay Northrup for a warm recovery facility. Why didn't Northrop understand these risks and failover to that facility during the maintenance? Has Northrup ever tested failing over the the recovery facility? Why does Northrup keep their database of backups on the same SAN as the primary copy of data? In a highly available hosting solution, archived logs should be written to the backup on a seperate SAN frequently during the day.

I have no confidence Northrup is giving the citizens the service we deserve.

Posted by: Fairfaxtechguy | September 14, 2010 10:20 AM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.

characters remaining

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company