The mystery of the missing Hotmail messages (updated)
Hotmail users who made a New Year's Resolution to clean up their inboxes got some unexpected help in that project over the weekend: A glitch in Microsoft's free Web mail service mysteriously wiped out the folders of thousands of users from Dec. 30 to Jan. 2.
As one of the first user reports on the company's Windows Live Solution Center forum put it, the experience was like having one's account reset back to day one: "I refreshed my browser and logged back in, only to see a 'Welcome to Hotmail' email and all of my previous emails missing."
One affected user, Mike Vicic, provided a more detailed description in an e-mail sent on the evening of Dec. 31. The Pasadena, Calif., engineer noted that the outage left users' contacts lists untouched, and he criticized Microsoft's slow response to the first complaints:
Techs were treating the posts on a case-by-case basis, and were trying to recover people's accounts one at a time with varying degrees of success.... [They] did not see the bigger, widespread issue until approximately 7 hours later when one tech replied: "We have already filed a report about this and we are currently investigating this issue. As I have checked the accounts, they are registered as new accounts."
A more helpful reply came in a separate thread on that forum on Jan. 2. A Hotmail developer reported that "we have identified the source of the issue, have restored email access to those who were effected." A second post Monday morning encouraged users who still couldn't get to their e-mail to file a report.
Vicic wrote that his inbox and other folders returned early Sunday morning, in the same state as before the abrupt sweep.
(Note that if you'd taken advantage of Hotmail's free mail downloading option -- also available at no charge on Gmail and AOL's Web mail service, but a $20/year upgrade at Yahoo's -- you would have still been able to read saved messages on your own computer.)
Microsoft posted a more public acknowledgment of the issue Monday afternoon on its Windows Team Blog. Wrote Chris Jones, a vice president for Windows Live services:
Beginning on December 30th we had an issue with Windows Live Hotmail that impacted 17,355 accounts. Customers impacted temporarily lost the contents of their mailbox through the course of mailbox load balancing between servers. We identified the root cause and restored mail to the impacted accounts as of yesterday evening, January 2nd. As with all incidents like this, we will fully investigate the cause and will take steps to prevent this from happening again. We're very sorry for the inconvenience this may have caused to you, our customers and partners.
That explanation makes it clear that only a small minority of Hotmail's 280 million active users suffered this outage. It does, however, leave room for a fair amount of speculation about just what pushed things off the rails. Were you among the Hotmail users hit with a blank inbox this weekend? If so, care to suggest your own theory in a comment?
Update, 1/7/11: You can stop theorizing. Microsoft posted a much more detailed explanation yesterday. In this case, Hotmail's standard diagnostic routine of creating test accounts to monitor the service's working went awry:
we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts. Please note that the email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed.
Fortunately, Microsoft's Mike Schackwitz writes, things should be fine now--"we had 100% recovery of existing email and folders in the affected accounts"--with one exception:
The only unfortunate exception to this statement is that, if you were affected by this incident and you didn't sign in to your account between the time of the incident and the time your account was restored, then any messages sent to your account during that time would have bounced.
Is your inbox still missing messages, or are things fully back to normal in your Hotmail account? We're still accepting comments on this, so let me know.
Posted by: qwerty1232 | January 4, 2011 10:18 AM | Report abuse
Posted by: wiredog | January 4, 2011 10:45 AM | Report abuse
Posted by: jerkhoff | January 4, 2011 11:42 AM | Report abuse
Posted by: mjohnston1 | January 4, 2011 12:20 PM | Report abuse
Posted by: Arlington4 | January 4, 2011 2:28 PM | Report abuse
Posted by: kaleidoscopeeyes | January 4, 2011 7:19 PM | Report abuse
Posted by: strade65 | January 4, 2011 8:36 PM | Report abuse