Network News

X My Profile
View More Activity
Posted at 3:59 AM ET, 03/ 1/2011

Google explains e-mail glitch, apologizes

By Hayley Tsukayama

A storage software update caused thousands of Gmail users to lose access to their e-mail Monday and Google was forced to go to its physical backup tapes to fully restore users' data.

Monday evening, Google's vice president for engineering and site reliability czar Ben Treynor wrote that most users' access had been restored. Treynor said that users' information was never erased, but that the bug did destroy several copies of the data. Google is well-known for its comprehensive disaster recovery plan, which came in handy Monday as the company had to leave the cloud to get the data.

By Tuesday morning, the Google Apps Status Dashboard gave Gmail the all-clear.

Fortune has a good explanation of what going to the tapes means, and estimates that getting all of Google's mail users backed up would take about 200,000 tapes measuring almost 2.5 miles high.

Did you lose access to your e-mail and has all your data been restored?

By Hayley Tsukayama  | March 1, 2011; 3:59 AM ET
Categories:  E-mail, Google  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: Oscar winners get statues and prestige, but losers get an invite to space
Next: Analysis says Bing overtakes Yahoo for the first time


Account still not restored. When I logged on Sunday morning, all email was missing and I'm still waiting for Google to restore my email account.

Posted by: jag05 | March 1, 2011 9:42 AM | Report abuse

Google has decided to isolate itself from its customers by denying its users any form of access in the event of a system disturbance. An air of arrogance prevails. Apologies are most annoying since they cannot recover opportunities lost because of Google's growing arrogance. I fear Google has become so gigantic that their attitude is the hell with them,the people like us who rely on email and browsers for information, contact with geographically distant friends and employment. Is is that Google has shifted priority to counting the money it is making from the ads they are selling? Have they forgotten the browser and email uses are those who see those ads? Perhaps it is excusable for the offshore support people to rely so heavily on "apologies" for company product deficiencies and poor product performance. They have no idea of the American culture or our annoyance over an empty "apology for the difficulties you are having today". It is unacceptable and naive for Google to think it can sustain a healthy customer base without maintaining a valid reliable product. This not the first time for Google to loose emails, drop service or disclose privacy information. It is not the first time this browser/email service has failed to meet the expectations and trust of their customer base. Google has lost touch with the public. Google has become a lesser used secondary source rather than primary provider for my computer needs.

Posted by: Capobiancof | March 1, 2011 10:29 AM | Report abuse

Access to my google account was restored (documents, picasa, etc), but not my gmail account.

Posted by: kellymcostello | March 1, 2011 10:34 AM | Report abuse

Hi,YOUTUBE (which is owned by GOOGLE), decided that MY YOUTUBE account MUST BE connected to a GOOGLE ACCOUNT. Fine, I said, so I began to follow their instructions. In the process I lost access to MY YOUTUBE ACCOUNT. I am in limbo. GOOGLE/YOUTUBE have no support, hey it is a free service. So, what to do? who to talk to? They have no 800 or a special email. Just FAQ web site. GOOGLE/YOUTUBE can you helpppppp

Posted by: iglooo101 | March 1, 2011 10:42 AM | Report abuse

Never knew there was an issue, but I wasn't around on Sunday. gmail is my primary email. Honestly, not too bad for free, eh? I'm shocked that they even put to tape the free accounts. I think Google is an amazing resource for free. I wonder if I can put the word FREE email in here one more time?


Posted by: tuFahb8u | March 1, 2011 10:45 AM | Report abuse

I recommend that disgruntled users demand a full refund from Google and take their business elsewhere. That ought to show them!

Posted by: BoteMan | March 1, 2011 10:45 AM | Report abuse

My google apps are back, email is not back.

Posted by: kwoodgr | March 1, 2011 11:01 AM | Report abuse

I am still not able to access my email. It's March 1, 8:00 a.m. PST here in Silicon Valley, well past the time that the contrite Google spokesman said everyone would be restored.

I know that those of us affected by the bug are a minuscule percentage of the Gmail user base - they certainly keep reminding us of that! But the communication from Google has been too late, too little and mostly inaccurate.

People are telling me there's no such thing as a free lunch. But, this should be a warning to others, especially large institutions, who are thinking of moving to (and paying for) their place in THE CLOUD!

Posted by: mmfamily | March 1, 2011 11:21 AM | Report abuse

I still can't access my email. I do have access to igoogle and picasa now which I didn't before. Waiting til tomorrow to start a barrage of email to google.

Posted by: jclarkson19 | March 1, 2011 12:06 PM | Report abuse

You get an email account with a big internet company, you are not paying anything, but you use this account for "important" stuff, but you cant be bothered to take the most trivial effort to back up your work. Wonder where we should lay the blame? Do you flinch when you look in the mirror?

Posted by: periboob | March 1, 2011 12:28 PM | Report abuse

Google has been slow for the last couple of months, this e-mail thing was only an addition.

Posted by: skr3211 | March 1, 2011 1:09 PM | Report abuse

We extensively rely on Google and honestly didn't notice any bottlenecks or interruptions.

I must admit I share in some frustration about customer service and the reality of a technology leader woefully short on resources tailored to deal with the human side of the equation.

We are fickle creatures and would rather deal with someone having a pulse rate and IQ over something with a clock rate and adaptive programing, perhaps Watson can contribute to a solution. It frustrates me that the only way to bring an issue before upper management is via FAX or U.S. mail, no email / web mail access at all.

Professionally I am neck deep and vested in technology, my view from the inside out is I would far rather see my crew doing what they do the best, applying cutting edge technology to unique undreamed of solutions rather than being distracted by a PR quagmire of holding hands.

Yes the human side needs to be addressed, hopefully this was the wake up call because if an issue falls outside of the knowledge base you end up lost in a unsolvable, inescapable auto attendant maze.

In overview, the entire concept of the internet is "the free exchange of knowledge" total interoperability between all platforms and data formats. Once we achieve that, no more deceit to save cooperate face, no more proprietary hidden secret holes in the wall, everybody will know where they are and how to fix them. Then and only then will we see security, reliability and consistent integrity of data raise exponentially.

The evolution we've seen just in the last 6 years gives me faith, divided factions within the tech world are beginning to work towards a common goal of cooperation. Somehow I believe this may be one of the best spinoffs from the Open Source model, as a cooperative community
a tremendous amount of progress can be accomplished and common ground can be found to unify efforts where division and deceit formerly flourished.

Yes, I really have faith that the upcoming generation will do more through applied technology, knowledge exchange and personal empowerment towards the unification of humanity
than the last 2000 years of failed attempts and authority through fear as practiced through political and Faith based structures.

Is the absence of email for 48 to 72 hours an inconvenience yes, is it a disaster no, this is why business and infrastructures have alternate email address on secondary hosts, many have secondary IP providers too, do you drive around without a spare tire?

Look at the miracles these folks have given us, the collected wisdom of eons, in your face with the movement of a finger.

The real challenge now, knowing the right questions to ask and what to do when you don't like the answers.

Posted by: rufus7 | March 1, 2011 1:28 PM | Report abuse

I recommend that disgruntled users demand a full refund from Google and take their business elsewhere. That ought to show them!

Posted by: BoteMan


And for all of you complaining about a lack of support, well just imagine how much it would cost Google to do that. They would have to employ half of India!!!!

Try finding 7.5g of free storage on the internet. Or a free, 99.999% spam free email account. Keep complaining about free things, it makes me want to vote republican!

Posted by: salparadise661 | March 1, 2011 1:47 PM | Report abuse

gmail = free
complaints about gmail = a lot
businesses or individuals relying on gmail = priceless (as in you can't put a price on whatever you want to call such behavior - stupid, reckless, ironic, etc.)

Posted by: rogernebel | March 1, 2011 2:24 PM | Report abuse

I never lost my account, but thankfully this event has ended the folly of me relying on Google for important data. Goodbye cloud!

Posted by: hoos3014 | March 1, 2011 2:26 PM | Report abuse

Google e-mail is free. When things are free, what do people expect? Fast, reliable service? Really? You do not get reliable service even when you pay for it!

Posted by: philly3 | March 1, 2011 2:32 PM | Report abuse

I was locked out of my account in December. According to google, someone had accessed my account in Mexico. I believe that "someone" was me, as I was there for UN climate change talks. It took me almost a month to restore my account and caused counteless headaches. I tried emails, only to be placed in unhelpful circular loops offering to reset my account password. I tried phone calls, only to be routed to a message that telephone email support is unavilable. I couldn't email them about technical support because I had to have a google account, and they locked me out of it. Eventually, a tech-savvy friend found a link to a form that could report a technical problem with gmail (one that did not require a gmail account--if I could still find it, I'd post it here for anyone locked out). I'm not sure whether filling out that form twice is what finally restored my account or whether they finally fixed their system, but the whole process was absurd.

Their email service may be free, and this experience has prompted me to not elect to used them for paid services (cell phone, etc.) until they get a responsive customer support system in place. A small percentage of the ad revenues generated from my email use could help pay for improved customer service and would make a big difference. Google should be better than this.

Posted by: yabbadabba1 | March 1, 2011 2:52 PM | Report abuse

We are a corporate Google Apps customer and have lost several of our accounts to this mess. Google has been completely unresponsive to our support requests. They've now informed us that any emails from the last 24 hours will not be recovered! Two of our deleted users are customer service managers...which means hundreds of lost emails from our customers.

Google doesn't even want to provide timely updates on progress, return phone calls (as promised), public explanations of the real impact. Instead Google and the media that follows them want to down play impact by showing that it is a fraction of one percent. All E-Mail systems have issues...however, the overselling of their redundancy and uptime is nullified by their poor customer support and communication.

Suggestion to Google: Put Customer Support and honesty in your contingency plans. Trying to sweep this HUGE problem for us affected under the rug, with no communication, is unacceptable. If I had a local E-Mail server, I would/could be giving my users timely and realistic updates.

Google's downtime should be reported accurately from the time the problem occurred to the time the last account was fully restored. I'd like the media to hold them accountable for these numbers.

Why does Google's App Dashboard not show todays status? Wouldn't that be a good place to communicate?

Ben Treynor should be stripped of the his arrogant title of Czar.

Posted by: baorion | March 1, 2011 3:02 PM | Report abuse

version 1.0

gmail = free
complaints about gmail = a lot
businesses or individuals relying on gmail = priceless (as in you can't put a price on whatever you want to call such behavior - stupid, reckless, ironic, etc.

version 1.1

google apps = free
complaints about google apps = a lot
businesses or individuals relying on google apps = priceless (as in you can't put a price on whatever you want to call such behavior - stupid, reckless, ironic, etc.

Posted by: rogernebel | March 1, 2011 3:24 PM | Report abuse

Google Apps for business is not FREE. Neither is buying a server(s), software licensing, employees to support them, cost for redundancy, time to administrate all of the infrustructre. The wasted time, reading pointless posts that add nothing to the conversation.

A small list of a portion of those rogernebel calls stupid, reckless...

Posted by: baorion | March 1, 2011 3:39 PM | Report abuse

I recommend that disgruntled users demand a full refund from Google and take their business elsewhere. That ought to show them!

Posted by: BoteMan | March 1, 2011 10:45 AM

How funny since most are free accounts ? If a person/company is using free gmail for commerical and professional services, maybe they should rethink that that model. Why would someone not imap or pop3 their mail as a backup ? These types of problems happen all the time, and if the user or company does not do their CYA, it's their fault. As for GMAIL paying customer, maybe you should think about another provider, or again, pop3/imap the account to minamize your losses when something happens.

Posted by: ropratt | March 1, 2011 4:18 PM | Report abuse

Been using GMail since the first week it came out, years ago. It has been a good experience and beats the crap out of Microsoft Outlook and it's derivatives. They have never lost my data, and never denied me access to it either. Other people have mentioned this already, but it obviously needs to be pounded home again and again, it's a free service. Any fool that doesn't back up whatever is important to him deserves whatever he gets.

Posted by: realneil | March 1, 2011 4:42 PM | Report abuse

"Been using GMail since the first week it came out, years ago. It has been a good experience and beats the crap out of Microsoft Outlook and it's derivatives."

Anyone care to guess what application you might use to provide a great local backup for your GMail files? Well, one option would be to download all of your server-based Gmail into (!) Microsoft Outlook. Microsoft bashing gets pretty old, especially when the poster doesn't know how it (or email) works.

Posted by: jpjmf | March 1, 2011 7:55 PM | Report abuse

Actually I don't use outlook because it's like being slowly pecked to death by chickens. IF and I say IF I was to want to D/L my over 9GB of GMail data to my PC, I would use Thunderbird. Or I might Use the Mac and use that there 'Apple stuff' to do it with.
Outlook and Outlook express has been the best way on Earth to get 'infected' for years. Stick your head back into the sand.

Then again, I could use the Ubuntu Linux box too!

Posted by: realneil | March 1, 2011 9:57 PM | Report abuse

Free is good but if it doesn't work, what's good about it. Did I forget to mention-free for us but profitable through ads for Google. No ads--no free service.

Posted by: calmsprings | March 2, 2011 10:36 AM | Report abuse

Free IS good and the GMail service works. I also don't use Internet Exploder and have found sweet refuge in Firefox. There is an add-on for Firefox that eliminates the display of ads on your browser. So I don't ever see Google's ads. NONE OF THEM
As I said before, my data has always been available to me and I have tons of it. (almost 19,000 messages and all of the attachments they came with) It amounts to almost 10 Gigabytes and I trust them to keep it safe for me. They always have.

Posted by: realneil | March 2, 2011 7:44 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.

characters remaining

RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company