The Checkout

Is User Generated Always Useful?

2007 is shaping up to be the battle of the health Web sites.

Nearly a decade after WebMD went online, advertising dollars are finally migrating to the Web in enough quantity that investors such as Time Warner and the Carlyle Group think health Web sites are a hot investment. So even though WebMD has entered the pop culture lexicon--as in when a character on "Law & Order" quips, "What? Did you read that on WebMD?"--the online encyclopedia of illness info has attracted competitors, including America Online co-founder Steve Case.

Case's Revolution Health Group yesterday launched a "preview" of, sort of a soft launch, more public than a Beta site but not as ready for prime time as an official one.

It's still early days for this site and whether it survives will depend a lot on whether it provides accurate, reliable information consumers can use. aims to be very Web 2.0, so it relies in good measure on, as Case puts it, "the wisdom of crowds." That means user-generated ratings of doctors, hospitals, and even treatments.

The idea is that once millions of users have posted their comments, the sum of all their thoughts will pan out into something useful.

I asked Robert Krughoff, president of the Center for the Study of Sciences to cruise and share his first impressions.

Why Krughoff? Well, in addition to starting Washington Consumers' Checkbook, Krughoff is a widely respected expert when it comes to the challenge of evaluating health care for average consumers. One of the central hurdles we all face when making health care choices is that independent, reliable information is hard to come by. We choose doctors with even less data than we choose restaurants with. We ask around and we don't even have published reviews to rely on.

Krughoff's group knows how hard this challenge is as well. It has published its own guides to doctors and hospitals since the 1970s. Krughoff is a stickler about sample size and sources of data, so it should be no surprise that he has some doubts about the Web 2.0 approach.

"I'm not sure their strategy is going to get them something people can use," he said of

He pointed out that when it comes to user-generated feedback on doctors, it's very hard to get a decent sample size especially via passive means such as asking people to post comments because, let's face it, no doctor has a million patients. And registration on the Revolution site is easy enough, he said, that someone could post positive comments about himself or herself--or bad-mouth a competitor--under five different e-mail addresses.

Regarding user comments about treatments, Krughoff worried that that approach is too "fuzzy" and might not contain sufficient clinical information to be reliable. After all, it's hard to evaluate what someone says about a treatment without knowing more about the person's symptoms and prior treatments.

Finally, when it comes to hospitals, he said there is other data out there that is potentially more useful than consumer comments, quantitative data such as death rates, infection rates and complication rates. The federal government collects that information and has been striving to make it user-friendly.

Insurance companies are another source of harder data on docs and hospitals. They have access to all that claims data, after all, and providers are just beginning to offer their members some of that wisdom. Consumer Reports also has a Web site that offers pricing information on prescription and generic drugs that is easy to use. You might want to check out those sources in addition to user-generated comments on sites such as

Moving beyond the wisdom of crowds, both Revolution Health and WebMD are offering services such as appointment scheduling and the ability to create a personal health record. Krughoff said these sounded like good ideas.

Revolution has also started what Case calls the "AAA" of health care, a membership service where people can telephone for help with insurance disputes, appointment scheduling and answers to health questions. Krughoff didn't get a chance to evaluate those.

Of course, this space is all about collective and individual wisdom, so if you've tried any of these sites or services, don't hold back. We want all the reviews we can get.

By Annys Shin |  January 23, 2007; 7:15 AM ET Consumer News
Previous: About That Car... | Next: Talk Less and Pay More


Please email us to report offensive comments.

It's "Center for the Study of SERVICES".

Posted by: The Cosmic Avenger | January 23, 2007 9:38 AM

I find user-generated material to be something which must always be viewed with suspicion. Take Wikipedia, for example. While much of the material is probably wholly innocuous and correct, use of information from it without the backing of prime sources is probably a bad idea (just look at the well-reported manipulations of the system).

In much the same vein, other sources should be mistrusted as well. After all, the online arena is an opt-in system. The only people who are going to be going in and rating any hospital are those people who have strong feelings- people with exceptionally good or bad experiences- NOT the average person.

Posted by: Castor | January 23, 2007 9:44 AM

I think user-contributed information is a great piece to add to the whole puzzle.

Posted by: gtowner | January 23, 2007 10:15 AM

A problem with user-generated content is that the comments will not be representative of everyone's experience. You're more likely to get responses from people who have had extreme, unexpected experiences. Someone who has a health treatment that works as expected and is doing just fine will probably not bother to report it. If that happens, then most of the comments will reflect extreme experiences -- even if most people have perfectly fine reactions.

Posted by: bdnc | January 23, 2007 10:26 AM

Valuable information about healthcare comes from multiple sources. One is the healthcare community. Another is people with health conditions and their friends and family.

Most diseases in the U.S. are treated based on an individual physician's skill, knowledge, and personal belief about what's most appropriate. Significant treatment variations exist in practice and physicians are all too often not fully aware of alternatives, side effects, etc.

The alternative - following a well-defined care map that reflects best practices and evidence-based medicine - is not nearly as prevalent, unfortunately.

So it's essential that consumers seek out broad information from other informed sources, not just their particular specialist or team.

There's no doubt that consumers value other patients as a critical source of information, both about treatment alternatives and options and perhaps even more importantly, implications and trade-offs of these alternatives.

As proof of this, numerous large free online health & wellness communities already exist for well-known diseases like cancer and diabetes as well as conditions that aren't household names. They're usually launched by people who are passionate about a particular health problem. And they virtually always offer important insights and perspectives and advice that are impossible to get from a healthcare team.

For example, the Diabetes Exercise & Sports Association offers guidance on adjusting medication and food intake for insulin-dependent diabetics who lead very active lives.

This is information that simply isn't known by most dietitians and physicians, who primarily deal with INactive diabetics and whose advice is often inadequate for extremely active patients.

In another example on a different site, a woman with breast cancer recently posted about the tradeoffs of different reconstruction options, given that she wants to preserve as much muscle function as possible. Her physicians were far less able to talk about the real-world implications than were other patients.

Many of these sites are remarkably accurate and balanced despite their putative lack of "experts."

For more on this phenomenon, take a look at Surowiecki's 2004 book "The Wisdom Of Crowds.

Posted by: Leslie Nolen | The Radial Group | January 23, 2007 11:06 AM

I'll have to agree wholeheartedly with Leslie Nolen. People who have particular health issues are probably more expert in giving advice than their physicians. Someone who has been through breast cancer or a difficult pregnancy would have more pratical information than a male physician who has never been through those things.

Has anyone ever heard of hemochromatosis? Did you know this condition, the most common genetic disease in the US, is not routinely tested for when you get a physical exam? Did you know you have to ask specifically for the blood test? There are at least two websites devoted to hemochromatosis offering a wealth of information, and yet you have to ask for the specific blood tests when you get a physical exam. It can easily be misdiagnosed and years of your life are wasted not knowing what the problem is. It's easily treated once it's detected.

Posted by: Southern Maryland | January 23, 2007 11:17 AM

If the options are user-generated content, some way to get information from ppl who share an experience, or no such information, I'll take the extra info any day. I can choose for myself which opinions are meaningful, which are dross. We all do that with everything we consume (or choose not to consume), and I'm glad they're introducing this forum. It's a place for discussion, not the answers to all your medical questions.

Posted by: Catherine | January 23, 2007 12:03 PM

I appreciate the efforts of Rev Health. I believe it's certainly a push in galvanzing transparency in our medical system. Most of us know the questions to ask our physicians and become familiar with our healthcare options by asking someone we know (friends, family, etc.). Why not have the opinions of a few extra ten, twenty, or a hundred?

Posted by: dcnative | January 23, 2007 1:35 PM

Hi, I work for Revolution Health and this is a great conversation! While I agree that it can take time to get a statistically significant sample size per individual rating, any information that can help with a decision can be meaningful. Most of the time, we may ask a few friends or family for a recommendation for a doctor, and we usually take that under advisement. I look at reading a few ratings about a provider in the same way. What if you just moved to the area and don't know many people? What if you need a specialist, but no one close to you has a good recommendation? This is some additional information to help you make an important decision about where you want to go for care.

As the number of ratings increase on Revolution Health, the more helpful this becomes. I encourage you to share your experience about your providers or hospitals at to help others who may be looking for a health care provider and want additional input.

In addition, Revolution Health does have additional quality information about hospitals like the type of information that Robert Krughoff mentioned. We have tried to make it easy for users to find and compare quality information such as number of patients, mortality rates, complication rates and more for many conditions and procedures. Over time we plan on incorporating more information to continue to help people make the best decision possible for their care.

If you would like to see a sample of the hospital quality information, please go to the following link:

Posted by: Amanda | January 23, 2007 2:02 PM

Sounds great! Nothing wrong with an Amazon like review system- so long as it is kept honest.

Posted by: Chris | January 23, 2007 2:27 PM

If only the nurses could rate the doctors. They know where the bodies are buried.

Posted by: Tomcat | January 23, 2007 3:32 PM

Hi. I work for a pharmaceutical company, and this is great! All we need to do to promote our drugs is hire legions of "patients" who will post descriptions of their "experiences."

Posted by: Pharmo-Man | January 23, 2007 3:37 PM

I think these ratings of doctors are probably better than no ratings, but most patients don't know anything about the quality of care they're getting. They know whether or not their doctor is nice. They they don't know if their doctor knows the latest on whatever disease they have, if he's particularly skilled at diagnosis, or if he's, say, killed half his patients. Or if, on the other hand, he's a total jerk, but a genius.

Now, for most people, nice is probably enough. I'm all for doctors not being jerks; I prefer my own doctors to be nice, and I don't have any particularly challenging diseases (that I know of). Probably anyone with an MD (or, ok, a B.A.) could handle most of the stuff I come in with. But if I needed to choose a doctor more carefully, I don't think a bunch of raters who enjoy their doctor's bedside manner would be the best source.

Unfortunately, there's no other source, though, so hey - like I said, ratings is better than no ratings. I agree with the commenter who said nurses should be able to rate doctors - they do indeed know where the bodies are buried. And I suppose there's nothing keeping nurses from logging onto Revolution Health and sharing their knowledge.

Posted by: h3 | January 24, 2007 10:27 AM

I want to clarify that in my remarks about Revolution Health, I was NOT arguing that consumers can learn nothing from posted comments about doctors or about diseases and treatments.

I fact, I think there is much to be learned about diseases and treatments from posted comments. This information will probably not give you a good estimate of likelihood of a treatment's effectiveness but you may learn something about a possible side effect, type of discomfort, social stigma, or other effect where it is important to know that it EVER happens even if you can't tell from the comments how frequent it is. You might also get ideas on ways to cope with a condition or with ill-effects of a treatment. If you have the time, more information on what others have experienced can rarely hurt.

One good website for this kind of insight is, where ordinary patients simply tell their stories related to a disease or treatment. For example, on the topic of bowel cancer, there are written comments and videos of patients talking about symptoms, tests, side effects, living with the disease, where they found support, the financial implications, and how they told their children.

When it comes to selecting doctors, my main caution is that getting a small number of comments on any given doctor may not give you an accurate guide to your likelihood of being satisfied or treated well. Doctors have good days/cases and bad ones and what one patient finds good communication another might not. The NUMBER of comments matters if you want to know your LIKELIHOOD that you will have good experiences with a doctor. A few bad or good comments just tell you that at least that many patients had that type of experience; you probably already knew--unless it is such an egregious experience that it should absolutely never happen--that the vast majority of doctors have some dissatisfied patients and some who are satisfied. You want to know about the odds for you, and that requires substantial numbers of patient ratings. At, we report ratings of hundreds of local physicians for whom we have at least 10 patient ratings; we are a lot happier when we have many more than 10 ratings to go on.

As with any online consumer comment website, of course, it is essential when deciding whether to rely on a doctor-ratng website to assess how likely it is that commenters have a stake in the rating. Could you just go on and make a comment yourself without any evidence that you had used the physician? More than one comment under different aliases or email addresses?

Of course, there are some types of comments that are useful even from only one commenter--that the doctor speaks Spanish, for example, or has office hours on Saturday, or doesn't communicate by e-mail.

One final point, I certainly wouldn't want to leave the impression that I don't believe patient ratings of physicians are important. Patients can tell us essential information about physicians not available from any other source. Not just information about satisfaction and pleasantness, but also about aspects of care that are at the heart of effective clinical care. Other patients can tell us for instance how well a physician listens--an essential for accurate diagnosis--or how well a doctor explain things--an essential for patients to know what to do, and why, to carry out a treatment plan.

Posted by: Robert Krughoff, Consumers' CHECKBOOK | January 24, 2007 7:46 PM

The comments to this entry are closed.


© 2010 The Washington Post Company