Northrop to pay $250,000 for study of August computer outage
Defense contractor Northrop Grumman has agreed to pay up to $250,000 to study why Virginia's state computer networks suffered a widespread outage at the end of August and it then took days to restore data to some state agencies, Chief Information Officer Sam Nixon told lawmakers during a briefing Monday morning.
Nixon told members of the Joint Legislative Audit and Review Commission that he was confident the figure would cover the costs of a full soup-to-nuts study of the computer outage.
An initial failure of a memory card in a data storage unit housed south of Richmond knocked some processes offline at 26 of 89 state agencies. The Department of Motor Vehicles had to turn away customers for almost a week.
Sam Abbate, a vice president of Northrop Grumman, which holds a $2.4 billion contract to run and modernize the state's computer network, told the delegates and senators that the company regrets the inconvenience to Virginians. "What should not have happened, happened," he said.
Sen. R. Edward Houck (D-Spotsylvania) suggested the company buy full page ads in state newspapers to apologize to residents--similar to ads the company took out in 2009 to defend itself against allegations of poor service. And Sen. Janet D. Howell (D-Fairfax) questioned how the failure could have happened when the state was promised multiple layers of backups to its new, modern system. "We were promised redundancy, and we're not getting it--and we're paying for it.," she said.
Nixon said much of the weeklong delay resulted from the complicated process of restoring and rebooting data corrupted during the initial failure. Computer experts had determined some data has been lost--notably 4,200 photos of customers who visited the Department of Motor Vehicles for new licenses on the afternoon of Aug. 25, when the memory card first failed. The DMV already has asked those customers to return and have their photos retaken.
Northrop Grumman meanwhile is paying a Minnesota company that ordinarily works with NASA in an effort to restore a much larger pool of corrupted DMV photos taken during the four days leading up to the outage, Nixon said Monday. He said that work is progressing well and he is hopeful that customers who visited the DMV during Aug. 21-24 ultimately will not need to have their pictures retaken.
"I'm cautiously optimist that won't be necessary," he said.
Lawmakers also discussed possible fines that could be imposed on Northrop Grumman as a result of the outage. Nixon said he believes the company can be assessed at least $100,000 for violating contract provisions that require that none of the state's 4,800 servers go down for more than four hours a month. The state also will likely be reimbursed some costs as agencies are refunded their payments for routine computer services that could not be performed by the company while computers were down.
Gov. Bob McDonnell (R)'s chief of staff also has asked agencies to compile a list of other costs incurred, including overtime payments to staff who worked around the clock to restore services. Nixon acknowledged, however, that Northrop can likely only be forced to pay for those costs if the company's negligence is found to have caused the outage. He said the company's contract is "vague" as to what kind of conduct would constitute negligence; the governor's office is in discussions with Attorney General Ken Cuccinelli on the point, he said.
The first indications of failure occurred when the data storage unit, a piece of hardware about the size of eight refrigerators, alerted its manufacturer, Massachusetts-based EMC Inc., that one of two memory cards in the device was experiencing trouble, he said.
A technician in the Chester facility, acting on instructions from EMC, then replaced the card. But as soon as the card was replaced, the memory devices failed altogether. Subsequent testing indicated problems with the memory card that had not been replaced, he said, raising the possibility that the wrong card was replaced or that both cards failed simultaneously.
September 13, 2010; 3:00 PM ET
Categories: Robert F. McDonnell , Rosalind Helderman
Save & Share: Previous: NRCC ad hits 'phony Perriello' on lobbyist donations
Next: Wilder launches blog with warning on federal spending
Posted by: Fairfaxtechguy | September 14, 2010 10:20 AM | Report abuse