Network News

X My Profile
View More Activity

Posted at 5:00 AM ET, 02/11/2011

Why our metrics for quality are not objective

By Valerie Strauss

This was written by Larry Cuban, a former high school social studies teacher (14 years, including seven at Cardozo and Roosevelt high schools in the District), district superintendent (seven years in Arlington, VA) and professor emeritus of education at Stanford University, where he has taught for 20 years. His latest book is "As Good As It Gets: What School Reform Brought to Austin." This appeared on his blog.

By Larry Cuban
Choosing the right metric to measure a medical therapy or school effectiveness is not a fact-filled, objective decision. It is subjective and packed with trade-offs.

Consider the metrics used since 1971 when president Richard Nixon signed legislation declaring a "War on Cancer.*"

In 1985, one researcher used the measure of how many lives were saved with chemotherapy and exams that detected the onset of cancer. He used the national cancer registry to determine how many Americans were diagnosed with cancer annually (about 1 million in 1985) and how many died of cancer in the same year (about 500,000). He then estimated how many cancer patients were “cured” (five years of remission) after heavy doses of chemotherapy alone (about 5,000) a year. He then estimated lives saved annually from chemotherapy given after surgery (35,000 to 40,000).

Total: Less than one out of 20 people diagnosed with cancer in a given year and less than one in ten of the total patients who died of cancer had benefited from screening and chemotherapy (pp. 227-229). According to this metric, the War on Cancer had hardly dented the problem. Other measures were used subsequently such as “age-adjusted mortality” that showed even worse numbers than “lives saved.” To medical researchers and practitioners, the nation was losing the War on Cancer.

Another medical researcher, however, showed that if the measure was changed from “cured” to “years of life saved,” cancer was being beaten. He calculated that if chemotherapy cured a 5 year-old child of leukemia, that child lived at least 65 more years (assuming age 70 as life expectancy); then if chemotherapy cured a 65 year-old man, he would have five more years, given the typical life span of 70 years. Other metrics could not detect any differences in the two cases. But using the measure of “years of life saved,” the researcher showed that therapies and detection had, indeed, made substantial, progress in fighting cancer.

What’s the point?

Even with a shower of statistics, measuring success in fighting a serious illness remains subjective. The counting of deaths–an objective process–could be done in at least three different ways leading to opposite conclusions on the effectiveness of the overall effort to curb cancer.

As some readers may have noted, I have said nothing about what has historically reduced the incidence of killer diseases (e.g., tuberculosis, cholera, typhus, scurvy, pellagra, lung cancer). No “magic bullet” pill or therapy reduced deaths but better sanitation, sewage, housing, nutrition and public health campaigns such as anti-smoking did. In effect, prevention reduced deaths from disease.

Turn now to schools where objective standardized tests are the chief metrics used to judge national, state, district, school, principal, teacher, and, yes, individual student effectiveness. International test results provoke media frenzies and blogosphere blather over declining quality of U.S. schools compared to Asian nations. Test results–supposedly precise numbers–are used to judge effectiveness and, simultaneously, hold educators and children accountable for results.

As with the War on Cancer, subjective choices are made constantly by testing experts in reading, math, and academic subjects about which items to include in tests (e.g.,multiple choice, open-ended questions, writing samples). Policymakers decide which tests to use (e.g., NAEP, state tests, teacher-made ones) and where to set cut-off scores to determine student passing or advanced standing.

Except for insiders in the testing industry, these subjective choices seldom leak out to the public or practitioners, much less to parents and students. And with a national climate for accountability dominant since the early 1980s, the quest for better teaching and learning has come to wholly rely upon a surrogate for quality–the standardized achievement test.With pressure to use student-test scores–”value-added measures”–for evaluating teachers and pay-4-performance plans, the illusion of objectivity has driven underground the subjective choices that top officials make.

After all, a test is a proxy for quality, not the real thing–it is the sizzle, not the steak. While the numbers produced from the tests have the patina of objectivity, the biases contained within the choices that psychometricians and policymakers make are less obvious but are nonetheless there (see Koretz_JHR).

Like the ever-continuing search for a medical “magic bullet” to cure cancer, Parkinson;s disease and obesity, prevention -- which has saved more lives in the past than any pill or therapy -- is often a footnote rather than the text.

Early interventions at preschool age by health and education authorities have clearly established (also see here and here) that the ill effects of poor health, poverty, and neglect can be reduced substantially. These preventive actions have far more payoff now and in the future than the restless search for a knock-’em-dead innovation or later interventions in high schools and colleges.

While tests, like medical metrics, are worthwhile as signals for intervention, they remain subjective measures of quality. Change the metric, the outcome changes.

*The content on metrics (and page numbers) used to determine success/failure of the War on Cancer come from Siddartha Mukherjee, "The Emperor of All Maladies: A Biography of Cancer" (2010)


Follow my blog every day by bookmarking And for admissions advice, college news and links to campus papers, please check out our Higher Education page at Bookmark it!

By Valerie Strauss  | February 11, 2011; 5:00 AM ET
Categories:  Guest Bloggers, Larry Cuban, Standardized Tests  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: Missing the point on Rhee
Next: Bedbugs at school: The new lice but worse


I must be missing something. It appears to me we have some gain/improvement in earlier start classes, but at what expense? At a time when school budgets are moving toward dust we are spending money for "glorified" day care? That is the position I read into this study.

This is the second time today I read an article writing about "intervention." As I stated in a separate article, this is a poor choice of words or a designed thought of educators to take control of children. Rather than agree our efforts since the 1950s have produced few results, we keep "intervening" earlier and earlier.

Let's stop discussing students as a result of education. Let's talk in terms of citizenship and "citizens." What are we getting for improved citizens from the effort? If we judge citizenship on fewer inmates, then sure we can say it works. If we are talking about citizens and drug use, or citizens and obesity, or some other comparitive study, what have we really gained?

Posted by: jbeeler | February 11, 2011 8:10 AM | Report abuse

I don't think of preschool as an intervention. I think of it as one of the most important years of education. I also think early childhood nutrition is extremely important. Without those things it is difficult to get going in life.

On the other hand, having preschool is not going to keep you out a gang in 8th grade. One of the reasons intervention programs "don't work" is that they are not always funded and after school and Saturday programs for kids are always the first things to be cut. Their funding basically depends on budget surplus and generosity.

Standardized testing is useful in comparing student outcomes on specific standards. But standardized tests are expensive. The booklets, the grading, etc. all comes out of school budgets. How much does it actually cost a district to offer those tests?

Posted by: georgia198305 | February 11, 2011 9:27 AM | Report abuse

Early interventions at preschool age by health and education authorities have clearly established (also see here and here) that the ill effects of poor health, poverty, and neglect can be reduced substantially. These preventive actions have far more payoff now and in the future than the restless search for a knock-’em-dead innovation or later interventions in high schools and colleges.
This is correct but the author misses the purpose for standardized testing in the United States since 2001.

The standardized tests are so that the Federal government does not have to spend more money in education while giving the illusion that they are improving public education.

This was the purpose of No Child Left Behind. Proclaim that every school must teach all the children in their school to a level of proficiency no matter the characteristic of the children, or that school will face penalties.

The whole purpose was to pretend that there was no difference in children in different schools in the United States. Accepting that there were no difference in schools there would be no need to spend additional monies for individual schools. Since all schools were the same with sameness in the children in the school it was feasible to mandate that school make all the children proficient.

Based on this you could claim that the amount of money needed to be spent for any school in the United States was x dollars. This meant there was no need for the Federal government to spend more than x dollars on any school.

The author, as so many in this country, has accepted the false idea that NCLB was meant to improve public education. The author is trying to offer logical argument from practices that have flowed from his acceptance of this false idea that NCLB was meant to improve education.

The idea that NCLB was meant to improve public education is a false and there is little point in attempting logical argument based on a false idea.

NCLB was based on saving the federal government money in public education while giving the illusion of improving public education.

NCLB is based upon the idea that there are no schools with children that are different from the children in any other school in the nation.

Therefore it is not necessary to spend money for preschool learning in any area of the country since all children that arrive at school are the same throughout the country.

Differences in scores for standardized testing are caused by poor school management and poor teachers since the children in schools are all the same.

Time for the author and other to understand that if you accept an absurdity such as NCLB there is no longer any purpose of logical arguments.

Posted by: bsallamack | February 11, 2011 11:11 PM | Report abuse

I wish that Valerie Strauss would stop having authors that attempt to give logical arguments based upon absurdities.

Standardized testing is based upon the absurdity of NCLB.

Imagine if in 2001 the President had said that we can mandate that every child in schools is proficient in football by 2014.

Americans would quickly see the absurdity of this.

Yet in 2001 the President actually did say that we can mandate that every child in schools will be proficient in learning by 2014 and Americans accepted this absurdity.

Features of our current public education system are absurd since they have grown out of the acceptance of the absurdity of NCLB.

Believe in an absurd idea and you start to accept anything that derives from that absurd idea.

Posted by: bsallamack | February 11, 2011 11:28 PM | Report abuse

". No “magic bullet” pill or therapy reduced deaths"

Has Mr. Cuban ever heard of antibiotics? Penicillin? It comes in pills. And the introduction of antibiotics saved millions of lives.

Posted by: educationobserver | February 12, 2011 10:24 AM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.

characters remaining

RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company