Network News

X My Profile
View More Activity
About this Blog   |   On Twitter   |   Follow us on Facebook   |   RSS Feeds RSS Feed
Posted at 1:20 PM ET, 02/15/2011

Computer audit blames hardware, human error for massive failure

By Anita Kumar
Anita Kumar

An unprecedented statewide computer outage in August was caused first by the failure of a key data storage system and then prolonged by human error when replacing the memory board, according to an auditreleased Tuesday by Gov. Robert F. McDonnell.

The independent review by Agilysys, a provider of information technology services, also found that there was a loss of backup data, a delay in restoring operations, and that Northrop Grumman failed to follow industry best practices.

The problem began Aug. 25 with the crash of a pair of three-year-old memory cards -- one was supposed to back up another. That led to the disruption of 13 percent of the state's executive branch computers, including the Virginia Department of Motor Vehicles, for more than a week.

McDonnell (R) has repeatedly pledged to straighten out the state's troubled $2.5 billion, 13-year computing contract with Northrup Grumman -- the largest contract in state history.

McDonnell spokesman Jeff Caldwell said the state has yet to determine how much the computer outage has cost the state or how much Northrop Grumman will be penalized for its mistakes.

"The disruption to our state computer system last August caused the Commonwealth to incur significant expenses,'' McDonnell said in a statement. "It impacted a broad array of state agencies, including some of the agencies our citizens most commonly interact with on a daily basis. Many Virginians were inconvenienced by this disruption and lost hours of their time in dealing with the outages. It was an unacceptable failure and one that cannot be allowed to reoccur."

Former governor Mark R. Warner (D) proposed consolidating the state's computer operations into one agency and overhauling the system after a legislative review in 2002 showed that the state's computer system was out of date and increasingly expensive to maintain. The General Assembly, then controlled by Republicans, approved.

The state accepted bids and selected Northrop Grumman, the giant Los Angeles-based defense company, for the contract, the largest of any kind in Virginia history. But the contract has been plagued by problems from the start, including missed deadlines and poor service. A blistering legislative audit released last year found that the computer system had caused problems at almost every state agency that uses computers.

"Northrop Grumman is committed to this partnership and dedicated to meeting our obligations with regard to the quality of the Commonwealth's infrastructure and the services we deliver,'' said Company spokeswoman Christy Whitman. "We have been working hard since the August 2010 outage to make the appropriate improvements to help avoid or mitigate similar disruptions. We are ready to engage in discussions with the Commonwealth about how the report recommendations could be best implemented."

By Anita Kumar  | February 15, 2011; 1:20 PM ET
Categories:  Anita Kumar, Robert F. McDonnell  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Virginia chooses 2012 presidential primary date, ending 'Potomac Primary' schedule
Next: Manassas officials approve plan to revitalize Georgetown South following last week's shooting spree

Comments

Should have gone with CSC who actually know how to run systems as these.

NG should stick to weapons.

Posted by: mikefromArlington | February 15, 2011 1:38 PM | Report abuse

LOL CSC really? I suppose they have the technology to predict when a memory board AND its back-up will fail at the same time whereas NG doesn't? This could have happened to any company with the contract, sorry you were outbid.

Posted by: thomasmb | February 16, 2011 7:10 AM | Report abuse

Anita:

You might want to take a closer look at this. The "audit" was done by Agilysys - an EMC partner (see http://www.agilysys.com/home/Partners/EMC/ ). This is an incredible conflict of interest - and there is a reasonable belief (based on the facts presented that an EMC hardware failure, not just human error, could also be a primary cause... but you don't see that in the report.

I am completely stunned that this obvious conflict of interest wasn't caught sooner. It's certainly not disclosed in the audit.

Posted by: InTheKnow6 | February 16, 2011 4:22 PM | Report abuse

Anita,

I think that is funny as well that Agilsys did the audit. Also, it goes to show that no technology is fail-proof. I do agree though that NG has made a made of the VA system.

Maybe the foundation of the article should have further called into question what EMC consistantly tries to do through sometimes questionable practices.

Posted by: concerned6 | February 17, 2011 3:02 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company