Network News

X My Profile
View More Activity

Posted at 5:00 AM ET, 12/ 6/2010

Do international test comparisons make sense?

By Valerie Strauss

Tomorrow we will learn the latest results from the Program for International Student Assessment, known as PISA and promoted as the most comprehensive study to test and compare student performance internationally.

Each time PISA, or other international test results are released, there is angst in the United States because American students aren’t ranked as high as Japan and Finland and Singapore and South Korea and a bunch of other countries.

Experts are quoted about how the United States is going to slip into oblivion if we can’t get these scores up, and other experts are quoted as saying that we have to speed up specific school reforms (the current ones in vogue involved high-stakes standardized testing, expanding charter schools, etc.) so that we can reclaim our rightful place at the top of these test result lists.

Expect to hear all of that this week and more.

So before all the hullabalo starts, it is a good time to look back at what the late, great social scientist Gerald Bracey wrote about international comparisons. Bracey was director of research, evaluation and testing for the Virginia Department of Education from 1977 to 1986, as well as a trained psychogist who was the leading critic of how today’s tests measure success. He authored numerous articles and books, including "Reading Educational Research: How to Avoid Getting Statistically Snookered."

Below are two separate writings, one a blogpost he wrote for The Washington Post blog x = why? and the other from his last Bracey Report on the Condition of Public Education. The report was jointly published in 2009, shortly after Bracey passed away, by the Education Policy Research Unit at Arizona State University and by the Education and the Public Interest Center at the University of Colorado. The whole report is worth rereading, but here’s part of what Bracey wrote on international comparisons of student test scores:

....Many critics cite the performance of American students on international comparisons of mathematics and science. The most often used comparison comes from rankings on the Programme for International Student Assessment (PISA), from the Organization for Economic Cooperation and Development (OECD). Most recently (2006), American students ranked 24th of 30 OECD nations in mathematics and 17th of 30 in science. Errors in the test booklets prevented the reporting scores for American students in reading.

It should be noted that these rankings are determined by nations’ average scores. Some researchers have suggested, however, that average score comparisons are not useful: even presuming that the tests have some meaning for future accomplishment, average students are not likely to be the leaders in fields of mathematics and science.

Those roles are more likely to fall to those scoring well. A publication from OECD itself observes that if one examines the number of highest-scoring students in science, the United States has 25% of all high-scoring students in the world (at least in “the world” as defined by the 58 nations taking part in the assessment—the 30 OECD nations and 28 “partner” countries). Among nations with high average scores, Japan accounted for 13% of the highest scorers, Korea 5%, Taipei 3%, Finland 1%, and Hong Kong 1%. Singapore did not participate.

The picture emerging from this highest-scorer comparison is far different than that suggested by the frequently cited national average comparisons; it is a picture that suggests many American schools are actually doing very well indeed.

Of course, the U.S. is much larger than these other countries and should be expected to produce larger numbers of successful students. But it is only when we look beyond the mean and consider the distribution of students and schools that we see the true picture. Students attending American schools run the gamut from excellent to poor. Well-resourced schools serving wealthy neighborhoods are showing excellent results. Poorly resourced schools serving low-income communities of color do far worse.


The second Bracey writing is from a blog that Post reporter Michael Chandler wrote while she was spending a year retaking high school math. In December 2008 she asked Bracey to write about the results of the just relased 2007 TIMMS test. Here’s what he wrote:

So the U. S. is not #1 in mathematics or science testing. So what?

So, very little.

First, comparing nations on average scores is a pretty silly idea. It’s like ranking runners based on average shoe size or evaluating the high school football team on the basis of how fast the average senior can run the 40-yard dash. Not much link to reality. What is likely much more important is how many high performers you have. On both TIMSS math and science, the U. S. has a much higher proportion of "advanced" scorers than the international median although the proportion is much smaller than in Asian nations.

This was not true on PISA, another international comparison that tests 15-year-olds. Only 1.5% of American students scored at the highest level compared to top performing New Zealand at 4% and second place Finland at 3.9%.

Yet the proportion of Americans at the highest level meant that 70,000 kids scored there compared to about 2,000 for New Zealand and Sweden. No one else even came close--Japan was second with about 33,000 top performers. These are the people who might end up creating leading edge technology in the future. Who cares if Singapore, with about the same population as the Washington Metro Area, and Hong Kong, with about twice that number, score high?

There aren’t many people there. (And, as journalist Fareed Zakariya found out, the Singapore kids fade as they become adults. More about that in a moment). The bad news is that the U. S., on PISA anyway, had many more students scoring at the lowest levels; these kids likely can’t compete for the good jobs in the country.

Second, test scores, at least average test scores, don’t seem to be related to anything important to a national economy. Japan’s kids have always done well, but the economy sank into the Pacific in 1990 and has never recovered.

The two Swiss-based organizations that rank nations on global competitiveness, the Institute for Management Development and the World Economic Forum, both rank the U. S. #1 and have for a number of years. The WEF examines 12 "pillars of competitiveness," only one of which is education. We do OK there, but we shine on innovation. Innovation is the only quality of competitiveness that does not show at some point diminishing returns. Building bigger and faster airplanes can only improve productivity so much.

Innovation has no such limits. When Zakariya asked the Singapore Minister of Education why his high-flying students faded in after-school years, the Minister cited creativity, ambition, and a willingness to challenge existing knowledge, all of which he thought American excelled in. But, as Bob Sternberg of Tufts University [he is now provost of Oklahoma State University] has pointed out, our obsession with standardized testing has produced one of the best instruments in the nation’s history for stifling creativity.

But really, does the fate of the nation rest on how well 9- and 13-year-olds bubble in answer sheets? I don’t think so. Neither does British economist, S. J. Prais. We look at the test scores and worry about the nation’s economic performance. Prais looks at the economic performance and worries about the validity of the test scores: "That the United States, the world’s top economic performing country, was found to have school attainments that are only middling casts fundamental doubts about the value and approach of these [international assessments]."

Third, even if comparisons of average test scores were a meaningful exercise, it only looks at one dimension--the supply side. Predictably, the results gave rise to calls for more spending on science instruction. This ignores the fact that we have more scientists and engineers than we can absorb. In one study, Lindsay Lowell of Georgetown University and Harold Salzman of the Urban Institute found that we mint three new engineers for every new job (this is from permanent residents and citizens, not foreigners).

More disturbing was the attrition rate. While educators fret over losing 50% of teachers in 5 years (and well they should), Lowell and Salzman found that engineering loses 65% in two years. Why? Low pay, lousy working conditions, little chance for advancement. American schools of engineering are dominated by foreigners because only people from third world nations can view our jobs as attractive. In fact, long-time science writer, Dan Greenberg, invented a new position for those emerging with Ph.D.’s: post-doc emeritus.

Schools are doing a great job on the supply side. Business and industry are doing a lousy job on the demand side. The oil industry, responding to increased demand for oil exploration raised the entry-level salaries for petroleum engineers by 30-60%. The number of students lining up to be petroleum engineers has doubled and enrollment at Texas Tech has increased sixfold.

As usual in these comparisons, Americans in low-poverty schools look very good, even in mathematics. They would be ranked third in the 4th grade (among 36 nations) 6th in the 8th grade (among 47 nations). This is important because while other developed nations have poor children, the U. S. has a much higher proportion and a much weaker safety net. When UNICEF studied poverty in 22 wealthy nations, the U. S. ranked 21st.

Finally, there are some curiosities that will take some time to analyze. Critics are fond of pointing to the Czech Republic as a nation that spend much less than we do on schools but scores much higher. Not this time. The Czech Republic has seen catastrophic drops in its math scores since 1995, 54 points in 4th grade, 63 points in 8th grade and is now well below the United States in both grades.

Forty-percent of [South] Koreans reached the highest level in 8th grade math. In PISA, only 1.1% did. Note that that is fewer than the 1.5% of American students at the highest level in PISA.

Then there are the gender differences: For some countries there are huge differences in 8th-grade mathematics---favoring females. Of the eight countries with the largest differences, only Thailand is not an Islamic nation. Does this reflect which girls get to go to school in these countries? I don’t know.

P. S. Overall the U. S. did pretty well in both subjects at both grades.


Follow my blog every day by bookmarking And for admissions advice, college news and links to campus papers, please check out our Higher Education page at Bookmark it!

By Valerie Strauss  | December 6, 2010; 5:00 AM ET
Categories:  Standardized Tests  | Tags:  gerald bracey, how u.s. students fare internationally, how u.s. students rank, international comparisons, pisa, standardized tests, timss, u.s. comparisons internationally  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: When teachers are their own worst enemies
Next: Willingham: Close to a magic bullet in education


Preemptive excuse-making?
And yes, how our average students are doing does matter. Unless we are willing to accept that we really only have a responsibility to make sure we get some kids at the head of the Bell Curve.

Posted by: patrickmattimore1 | December 6, 2010 5:40 AM | Report abuse

If we spent half as much time working with how those scores equate to life and our GDP we would see that parts of this story is spot on.

Yes we should want our students to perform better, but at which point?

Posted by: educ8er | December 6, 2010 7:20 AM | Report abuse


By definition, their can only be some kids ahead of the bell curve. Now whether or not the bell curve is an accurate way to talk about the distribution of skills or knowledge is another matter.

Posted by: stevendphoto | December 6, 2010 8:52 AM | Report abuse

their/there. oops.

Posted by: stevendphoto | December 6, 2010 8:53 AM | Report abuse

To true Steven. I get tickled when I hear people say they want all students above average!

Posted by: educ8er | December 6, 2010 8:57 AM | Report abuse

"... I get tickled when I hear people say they want all students above average! ..."


I often wonder if some who post on 'The Answer Sheet' are from Lake Wobegon, "where all the women are strong, all the men are good looking, and all the children are above average."

Gerald Bracey's work is both entertaining and serious, which makes his passing last year so sad. His critiques and analysis of public education policy resonate with experienced educators, as he explained how testing data could be properly used to gain perspective of the condition of schools and communities, and how data is misused to promote political and business objectives.

Please consider, if you will, our nation's most touted and current pop-education program, 'Race-to-the-Top'. Only in Lake Wobegon, Minnesota do the schools have any reasonable chance to be the best. And the rest of us will be fighting for second place and below.

When I was in university, my friends (me too) really hated "grade-grubbers" and 'brown-nosers" as they were obsessed with class rank and usually dumber and less interested in truth.

The real value of an education is when it develops the individual and advances community knowledge, which leads to the type of creative and innovative work that has been at the core of American leadership. Not a bunch of score obsessed bean-counters without a basic understanding of how we actually inspire people to learn.

Everyone in position to form education policy and develop curriculum should read Gerald Bracey, and we should expect them to ...

Posted by: AGAAIA | December 6, 2010 11:38 AM | Report abuse

Bracey states, "Well-resourced schools serving wealthy neighborhoods are showing excellent results. Poorly resourced schools serving low-income communities of color do far worse."

Closing of the gap....consider the cumulative effects of many variables. Look at fluoride, known to negatively impact IQ (1). Furthermore, ".... fluoride consistently increases BPb [blood lead levels] and calcified tissues Pb concentrations in animals exposed to low levels of lead and suggest that a biological effect not yet recognized may underlie the epidemiological association between increased BPb lead levels in children living in water-fluoridated communities" (2). Black children (and Hispanics) have higher levels of lead in blood (3). Formula fed infants (more common among Blacks) are at greater risk for exceeding "safe" limits of flouride (4).





Some enlightened countries reject fluoridation of drinking water - interesting to correlate with international testing stats. The link below shows fluoridation info by country:

Posted by: shadwell1 | December 6, 2010 12:43 PM | Report abuse

I agree with most of Gerald Bracey's critique of international test comparisons. How much we miss his voice!But I take his comments not so much as a critique of the assessments themselves as a critique of how politicians spin and cherry-pick the results. And, by implication, he makes a critique of lazy education reporting(the Answer Sheet emphatically excepted)that fails to dig beneath the spin. One wonders if many education reporters ever read beyond the executive summaries of the tests. Irwin Kirsch, whose expertise contributed to the design of several international adult and K-12 assessments such as IALS, PISA, and PIAAC, has repeatedly tried to cut through the government and media spin with thoughtful analyses of the results he helped to generate. See for example his "Perfect Storm" which interprets US adult literacy results in the light of growing income inequality, child poverty, and other factors that Bracey and others have consistently identified.

Posted by: jstrucker | December 6, 2010 12:56 PM | Report abuse

Bracey attempted to justify our poor results on international tests by pointing out that we nevertheless had many thousands of kids who were top-notch students, even though those thousands represented only a tiny percentage of the American students tested.
In fact, relative to the higher performing countries, we have low percentages of students at the top of the Bell Curve and high percentages at the low end. Bracey could just as easily have pointed out that our absolute numbers of low-performing students dwarf the absolute numbers coming from the high-performers.

Posted by: patrickmattimore1 | December 6, 2010 10:55 PM | Report abuse

Well, if there was NO determinable linkage between "above average" performing students and "above average" performing nations, then we wouldn't have anything to worry about.

BUT, there DOES appear to be some kind of correlation. And a more important question, does a downward trend with student performance forecast a downward trend in the performance of a nation?

Posted by: wred | December 7, 2010 2:03 AM | Report abuse

Oh, oh. Results are in. U.S. decidedly average. President calls it Sputnik Challenge. Sec. of Education says it's time to wake up. More likely result is striking up another chorus on the fiddles and listening to the voices that continue to insist the kids are alright.
My suggestions. Emulate China. Adopt more Confucian programs in our elementary schools. Send hundreds of teachers to Shanghai to study what they are doing. Arrange thousands of student exchanges with Chinese schools so that our students can get a taste of what it is like to really work.
Forget Sputnik. That was about stimulating our best and brightest. What the U.S. needs is a program for the middle. Look at China. They not only created an economic miracle in the past 30 years; they are in the process of creating an educational one as well. Wake up and smell the tea America.

Posted by: patrickmattimore1 | December 7, 2010 7:11 AM | Report abuse

Most people in this country are as fat and lazy mentally as they are physically. It's a lot of work but if you sit down every night with your kid and help him study, the kid will completely destroy these standardized tests. The vast majority of US parents are too absorbed in their own self interests to do that so the kids in the US that are succeeding are pretty much the self motivators only.

Posted by: peterg73 | December 7, 2010 12:26 PM | Report abuse

The comments to this entry are closed.

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company