Guest Blogger: What International Math Tests Really Say

Our guest blogger today is Gerald Bracey, a trained psychogist, author of "Reading Educational Research: How to Avoid Getting Statistically Snookered," and a critic of how today's tests measure success. He was director of research, evaluation and testing for the Virginia Department of Education from 1977 to 1986.

I asked him for his response to the results of the 2007 TIMMS test, released yesterday. Read today's story in the Washington Post.

So the U. S. is not #1 in mathematics or science testing. So what?

So, very little.

First, comparing nations on average scores is a pretty silly idea. It's like ranking runners based on average shoe size or evaluating the high school football team on the basis of how fast the average senior can run the 40-yard dash. Not much link to reality. What is likely much more important is how many high performers you have. On both TIMSS math and science, the U. S. has a much higher proportion of "advanced" scorers than the international median although the proportion is much smaller than in Asian nations.

Second, test scores, at least average test scores, don't seem to be related to anything important to a national economy. Japan's kids have always done well, but the economy sank into the Pacific in 1990 and has never recovered. The two Swiss-based organizations that rank nations on global competitiveness, the Institute for Management Development and the World Economic Forum, both rank the U. S. #1 and have for a number of years. The WEF examines 12 "pillars of competitiveness," only one of which is education. We do OK there, but we shine on innovation.

Innovation is the only quality of competitiveness that does not show at some point diminishing returns. Building bigger and faster airplanes can only improve productivity so much. Innovation has no such limits. When journalist Fareed Zakaria asked the Singapore Minister of Education why his high-flying students faded in after-school years, the Minister cited creativity, ambition, and a willingness to challenge existing knowledge, all of which he thought Americans excelled in.

But, as Bob Sternberg of Tufts University has pointed out, our obsession with standardized testing has produced one of the best instruments in the nation's history for stifling creativity.

But really, does the fate of the nation rest on how well 9- and 13-year-olds bubble in answer sheets? I don't think so. Neither does British economist, S. J. Prais. We look at the test scores and worry about the nation's economic performance. Prais looks at the economic performance and worries about the validity of the test scores: "That the United States, the world's top economic performing country, was found to have school attainments that are only middling casts fundamental doubts about the value and approach of these [international assessments]."

Third, even if comparisons of average test scores were a meaningful exercise, it only looks at one dimension -- the supply side. Predictably, the results gave rise to calls for more spending on science instruction. This ignores the fact that we have more scientists and engineers than we can absorb.

In one study, Lindsay Lowell of Georgetown University and Harold Salzman of the Urban Institute found that we mint three new engineers for every new job (this is from permanent residents and citizens, not foreigners). More disturbing was the attrition rate. While educators fret over losing 50% of teachers in 5 years (and well they should), Lowell and Salzman found that engineering loses 65% in two years.

Why? Low pay, lousy working conditions, little chance for advancement. American schools of engineering are dominated by foreigners because only people from third world nations can view our jobs as attractive. In fact, long-time science writer, Dan Greenberg, invented a new position for those emerging with Ph.D.'s: post-doc emeritus.

Schools are doing a great job on the supply side. Business and industry are doing a lousy job on the demand side. The oil industry, responding to increased demand for oil exploration raised the entry-level salaries for petroleum engineers by 30-60%. The number of petroleum engineers has doubled and enrollment at Texas Tech have increased sixfold.

As usual in these comparisons, Americans in low-poverty schools look very good, even in mathematics. They would be ranked third in the 4th grade (among 36 nations) 6th in the 8th grade (among 47 nations). This is important because while other developed nations have poor children, the U.S. has a much higher proportion and a much weaker safety net. When UNICEF studied poverty in 22 wealthy nations, the U. S. ranked 21 st in the proportion of kids living in poverty.

Finally, there are some curiosities that will take some time to analyze. Critics are fond of pointing to the Czech Republic as a nation that spend much less than we do on schools but scores much higher. Not this time. The Czech Republic has seen catastrophic drops in its math scores since 1995, 54 points in 4th grade, 63 points in 8th grade and is now well below the United States in both grades.

Then there are the gender differences: For some countries there are huge differences in 8th-grade mathematics -- favoring females. Of the eight countries with the largest differences, only Thailand is not an Islamic nation. Does this reflect which girls get to go to school in these countries? I don't know.

By Michael Alison Chandler  |  December 9, 2008; 4:07 PM ET  | Category:  Math Around the World
Previous: International Math Test Results | Next: Friday Quiz, Take 3


While I agree with what was said in the article, I do take exception to the idea that certain localized standard testing such as math or chemistry at the high school level is not with out benefits. I being an Army brat, attending numerous middle and high schools through out the world, would have never gotten into college if it weren’t for the cramming I had to do for the NY state “regents” exams during my junior year at a long Island high school back in the days.

Posted by: tonyholst | December 10, 2008 11:44 AM | Report abuse

There are some good points here, but other parts are confused. You write:

"Second, test scores, at least average test scores, don't seem to be related to anything important to a national economy. Japan's kids have always done well, but the economy sank into the Pacific in 1990 and has never recovered."

First of all, Japan has essentially the same GDP per capita as UK, France, and Germany. More fundamentally, data shows there is a strong but imperfect link between average IQ test scores (which will be related to TIMSS) and per capita GDP (see "IQ and the Wealth of Nations"). You are however correct to note the importance for GDP of the abnormally percentage of high performers in the US (due to Jews, plus the top-notch immigrants we attract from around the world). The same point about this "smart fraction" is elaborated upon in this article:

Your demand-side point is well taken. Of course Bill Gates and all the other leaders of technology companies use data like TIMSS to argue that our educational system doesn't produce enough qualified engineers and so they need to be given expanded H-1B visa allotments so they can import engineers from India, etc. In fact, as you suggest, we have a glut rather than a shortage and the motivation of Gates et al. is really to further drive down wages.

Posted by: qaz1231 | December 10, 2008 11:55 AM | Report abuse

At last, someone with some common sense is allowed to write an article! Mr. Bracey is correct about engineers and tech workers, too. I am an engineer (and, also, a theoretical mathematician) who has worked in hi-tech for more than 30 years. I have nearly 50 patents, most of which are current (things like optical fiber networking, internet security, encoding and decoding information, cryptography, etc. I have been laid off three times since 2000, in every case being displaced by a cheap young indian H1-B guest worker. The last time I found out why. It wasn't just a higher salary, it was benefits. Most hi tech employers (actually, most large U.S. firms) are "self insured". They set aside a pot of money to cover medical expenses for their employees and dependents. If that money isn't used, the executives reward themselves with big fat bonuses. If medical expenses are greater than that medical account, the extra comes out of company profits. The H1-B guest workers are invariably single and young, so their medical expenses are cheap and they have no dependents to be worried about. (In my case, my youngest son was sick and had high medical costs. So, my employer canned me. That's right, *all* self insured employers comb the medical claims for employees and their dependents. A risk score, judging potential future costs, is assigned to an employee based on their costs and higher dependent's costs -- all assigned to an employee. If those costs exceed a set amount, or if the risk score is higher than a set amount, that employee is moved to a layoff list.) The same holds true for other benefits like vacation, leaves of all sorts, educational costs. In being allowed to comb the world, looking for the cheapest production and labor costs, U.S. companies are not just inhuman, they are detrimental to families. It's time to end the H1-B visa anyways. Since 2003, we have issued nearly three times more H1-B visa than there are hi-tech job openings. The result has been the wholesale dismissal of more than 3 million U.S. engineers and computer programmers, more than twice the jobs the $34 billion bailout of the automotive industry is supposed to save. Ending the H1-B visa wouldn't cost the taxpayer one dime, either. It would actually save the taxpayer's money. The ONLY reason this disgraceful program exists is because politician's receive huge contributions from the ruthless swine that run those corporations.

Posted by: mibrooks27 | December 10, 2008 12:35 PM | Report abuse

To poster milbrooks27:

Termination for medical expenses sounds pretty spurious to me. It could potentially be part of number of classes of wrongful termination: either a violation of public policy, violation of anti-discrimination (Americans with disabilities act, or age discrimination), or a violation of covenant of good-faith dealings.

If I were you, I'd get a lawyer.

Posted by: girish_joshi | December 10, 2008 1:32 PM | Report abuse

Good questions are raised in this opinion. But you have the ingredients for an important contradiction. While we have many high performers on the tests, our averages are low, implying many low performers. You also point out that we have a very high poverty rate for a wealthy country. I see a link there in the number of low performers and poverty. It could be based upon poor schools, poor environment, low hope, ... who knows? But the implication is that the test scores are telling us something important.

Posted by: lenupinnh | December 10, 2008 1:47 PM | Report abuse

I think what makes the most sense is a path to green cards for graduates of US schools (perhaps bachelors or masters level) if they pass medical and security checks. I think this is a better idea than depressing wages with foreign born workers. What we know about foreign born graduates of US schools is they are highly educated, highly motivated, have spent at least four years being acclimated to the US and are typically US friendly. I think they are a good investment because:

1. It makes them stakeholders in America.

2. It allows them to contribute to the economy in any way that seems sensible (thus taking the pressure off the wages of the few fields that take H1B) and also allowing the free market to operate. Their best contribution to the US economy might not be as a post-doc.

3. They will be net contributors to the economy so they are a good bet.

4. They've often contributed several thousand dollars to the US economy (40,000 * 4 years) by graduation. This is not to say they deserve something because they gave money but it shows that many often have significant resources although the stereotype is that they are all poor.

5. In absolute terms, they are not a large number but they are a significant number if you stack up their economically relevant demographic data against the groups that do the best in America.

6. It also increases the number of Americans that like sciences and are good at sciences. Such people will bring their love of science to their children and to their neighborhoods. One can imagine somthing like the effect that German-Jewish immigrants had on American intellectual culture after the world war.

Posted by: mathlete | December 10, 2008 1:58 PM | Report abuse


"But the implication is that the test scores are telling us something important"

At the risk of sounding glib, I think it tells us that America is a capitalist country with little in the way of social safety nets. The mechanisms of capitalism allocate the most undesirable conditions to those with the least purchasing power. I am not saying this as a slander against capitalism. It is not surprising that complex systems would have both good and bad outcomes.

Posted by: mathlete | December 10, 2008 2:08 PM | Report abuse

While it may be true that national test score averages don't correspond well with economic activity, it only suggests that dominant

factor in economic activity is not aptitude.

Chiefly I would argue that our country's economic dominance comes from policies regarding capital which make it possible to

nurture an innovative environment -- a factor completely independent from the academic aptitude of our people.

I would argue further that American innovation comes from the rewards afforded by such policies, rather than any particular

distinction in our academic system.

A point that the author doesn't seem to take is that math and science testing are used as a proxy for an educational system as a

whole (1. because they are easier to measure objectively, and 2. because critical reasoning ability translates will between fields).

By that argument, it isn't really relevant if there are enough technical jobs or not - presumably someone who excels in reasoning (as

measured by such tests) will be better at whatever job they take, no matter how technical the task.

I believe, the decline in our testing results indicates a decline in the quantity of "quality thinkers" we are producing nationally. Would the author have us believe we have "enough"?

Posted by: girish_joshi | December 10, 2008 2:11 PM | Report abuse

Blaming standardization for stifling creativity is an excuse for pea-brained dunces who can't handle both.

And creativity without the ability to learn doesn't exist. In order to have creativity in the first place, you need to have a basic IQ.

The best high schools in this country yield the most innovative students of all, partly b/c standardized tests are such a laughable breeze that creativity is a natural byproduct of smart people who don't complain like this writer.

And the prejudice against asians here is appallingly judgmental. There are many motivations for why some asians seek happiness through success apart from what this writer assumes they should have. And then to blame their academic process, when it's such a small part of their culture, is ridiculously harsh and ignorant. Innovation is inherent in many asian cultures, whether they choose to exercise it or not in the careers they value.

Posted by: garygfamily | December 10, 2008 2:36 PM | Report abuse

This article is ridiculously wrong. First of all, trying to connect the economy to the test scores is wrongheaded: do you think Saudi Arabia's economy has much to do with test scores? Just because there isn't a correlation there, doesn't mean test scores aren't important for measuring the potential output of more advanced economies.

Look at the our economy over the past 20 years the inflated GDP numbers from people trading bubble priced homes and equities. We are already over the precipice, and only people that don't want to look down can't see how far we've fallen. Any economic leadership we happen to maintain is a hangover from the times when we were a technology exporting powerhouse.

"What is likely much more important is how many high performers you have."
Which we are way behind the Asian countries at all levels.

"Japan's kids have always done well, but the economy sank into the Pacific in 1990 and has never recovered."
The Japanese economy didn't so much sink as it had its bubble pricked in the 90s due to over investment and took a long time to return to normalcy. They are engineering and innovating better than us in most areas. You can't innovate without a understanding of what the current leading edge is. Knowing how to read and add doesn't make you less creative.

"This ignores the fact that we have more scientists and engineers than we can absorb. "
That is simply not true. The whole point of innovation is to make this something more than a zero sum game. More scientists and engineers offer more opportunities to create. It is the very lack of supply that has driven up the cost to hire engineers.

I realize the service economy of the past two decades has made it harder to see, but much of the world has been subsidizing our lifestyles with their savings. When they stop doing that, maybe next year, things are really going to change.

Please, let's cut back on the lawyers and financiers and make things that will take us into the future. 100% scholarships in the areas where we have needs.

Posted by: staticvars | December 10, 2008 3:07 PM | Report abuse

Lots of political correctness and justification in minimizing the race of the academic high performers. The assumption being that anyone can score as high as Asians if they just study hard enough.

In most global studies, Asian nations such as Singapore, consistently outperform other nations (including the US) in math and science. One could justify this by the work/study ethic of the Singaporeans, specifically Chinese.

Spend some time in a typical Singapore school and you'll notice vast differences between the US system. Well behaved and prepared kids and much less teacher/student discussion during class. US kids, and teachers, expect lots of back and forth. Not so much in SG. Also very common in SG for kids to take "tuition" which we would call tutoring. Very strong work/study ethic.

However (an inconvenient fact) in the US, Asians have the highest average education and income. Asians also outperform all other races in the US in math and science regardless of socioeconomic level. So, even in America, in the hood, 3rd generation American Asian kids are acing their SATs.

While we readily accept a racial basis for disease and athletic performance (otherwise, there's a vast racist conspiracy in the NFL and NBA), we're not ready to consider a racial basis for intelligence/academic performance.

Now take a breath, I'm not saying that whites can't be MVP or blacks can't win the Field medal, merely that as a society we shouldn't expect equal performance from all.

Posted by: HypocrisyistheOpiateoftheLeft | December 10, 2008 3:12 PM | Report abuse

Certainly worrying about fourth grade math scores is a joke. But I suspect the analysis of the situation for engineers expressed in this article is not much better. As jobs go, top grade software engineering jobs are still pretty good jobs. Most technology companies seem to feel the need to staff many of those jobs with foreigners. While that feeling might be partially misguided, the reality is that there is a wide variation in performance among software engineers. Whether that variation has any relationship to the effectiveness of our schools is certainly highly questionable. But we should make more effort to be sure that our students get the best quality mathematics education. We should also work more effectively at understanding the effectiveness of teaching techniques and the limitations in applying them.
Mathematics edication is almost completely about learning a well defined set of basic skills. Most of the learning process is through exercises where students learn by doing. It should be possible to completely automate the process for teaching these skills and ensure that all students have top quality instruction that can adapt itself to particular student's needs. This kind of approach will not solve all problems. But we will not know what is possible until we make a real effort to get beyond nineteenth century educational techniques and see what it is possible to do with twenty-first century technology.

Posted by: dnjake | December 10, 2008 3:27 PM | Report abuse

Its about time somebody speaks the truth on this and cites a reputable study instead of the "American's are stupid" and "we need more engineers" nonsense we usually hear out of Wapo and other mainstream newspapers.

staticvars wrote:
"The whole point of innovation is to make this something more than a zero sum game. More scientists and engineers offer more opportunities to create. It is the very lack of supply that has driven up the cost to hire engineers."

The only thing a surplus of engineering graduates is going to create is more applications to graduate school. With few exceptions, new grads don't have the knowledge and are too debt ridden to start innovative companies. If nobody wants to hire them, they have to go to graduate school or leave the field. And where are you located that the cost to hire engineers has been "driven up"? Engineer salaries nationally have barely kept up with inflation. Maybe some indirect cost of hiring an engineer (the corporate recruiter's bar tab maybe?) has gone up, but salaries haven't. There's nothing to indicate a lack of supply which the cited Lowell and Salzman document should exhibit. Its 51 pages long, so all I've read of it so far is the abstract.

Posted by: bill3 | December 10, 2008 4:56 PM | Report abuse

HypocrisyistheOpiateoftheLeft wrote:
"While we readily accept a racial basis for disease and athletic performance (otherwise, there's a vast racist conspiracy in the NFL and NBA), we're not ready to consider a racial basis for intelligence/academic performance."

I'll agree with your assessment. Its no surprise to anyone paying attention that certain races value education more than others and perform better academically than others. That is pretty much the reason I'm against quotas and affirmative action. Those things try to legislate something which naturally won't exist. And so what if it doesn't. Let the asians work their butts off and spend their life's savings on five square meters of land in Tokyo. We've got it pretty good here and we don't have to go through some bootcamp-like schooling to get it. Life is about a lot more than studying and work. And this country at least holds its own on innovation. The day that we don't, maybe we make science as appealing a career field to our best students as business or law so that we can stay tops in the world, but right now I wouldn't lose any sleep over it.

Posted by: bill3 | December 10, 2008 5:21 PM | Report abuse

One factor that no one has mentioned is that Asian languages make it much easier for children who speak them to learn math. The number naming system makes the base 10 connection explicit. For example, while in English we say "forty five" while the Chinese word translates to "four ten five". This helps children to quickly learn mental math.

In our family's homeschool, I'm using a program that incorporates the Asian way of number naming and it really does help my daughter with mental calculation. I think it should be a standard part of the math curriculum here in the U.S.

Posted by: CrimsonWife | December 10, 2008 6:05 PM | Report abuse

This post is correct. As a former American software and current lawyer, I can assure readers of this comment that no American would become a software engineer unless he is insane. Why compete with hundreds of thousands of Indians entering the job market every year on H1-B visas every year when there are lower-hanging fruit to grab? When you see someone like Bill Gates testify before Congress that America needs to further open its job markets to foreign software developers, I encourage you to throw fruit at your TV.

Posted by: georgejones5 | December 10, 2008 11:46 PM | Report abuse

I am by no means an expert but I wanted to google around to see what the accepted economic scholarship was on H1B effects on wages in the IT industry. Understandably, some people get very emotional about the topic. It's surprisingly hard to find anything peer-reviewed.

The best I could find was this Economic Review from the Federal Bank of Atlanta:

which says that the effect on contemporaneous wages is at most too small to be measured.

Posted by: mathlete | December 11, 2008 9:09 AM | Report abuse


I heard that law in general had a very unpleasant working environment, very similar to what you described about IT. Is that true?

Here is one article:

Or just type "Don't go to law school" in google. There are many, many such articles.

If so, could it be there are issues with professional wages beyond immigration? (Personally, I think there might be issues with globalization and automation to consider. I also think there is a risk that high local wages might accelerate both these processes, the way that high oil prices can accelerate the adoption of energy saving measures.)

Posted by: mathlete | December 11, 2008 9:20 AM | Report abuse

I was impressed by a lecture economist Lester Thurow delivered at my workplace nearly 20 years ago. Back then, he was sounding the alarm that America needed to improve the education system. Why? Because when a German auto manufacturer opened a factory in the U.S., they had to import German high school students to do the tasks that the adult American workers could not begin to learn, given their educational preparation.
It helps that Mr. Thurow is a good speaker and a good writer.
Read Thurow's 'Building Wealth' in the Atlantic Monthly, June, 1999, (; by part 4 of the article, he stresses the need for U.S. students to improve their science and math skills.
Why? Just because they must.
It's shocking that anyone can say that there isn't a problem with the way U.S. children are educated in the sciences and math. The statement 'test scores, at least average test scores, don't seem to be related to anything important to a national economy' takes a very narrow view of what test scores mean.
Test scores don't mean money. But they do mean the future, and with the future there is money. What lazy people we are not to see the need to become competitive!
We aren't creative in a way that gives us an edge when learning. We are 'creative' about rewarding opportunity economically. It is the business environment where the carrot (reward) is 'Show me something new and I'll make you rich.'
First came the stick (punishment), however, when the student was required to work hard mastering skills. Other countries do that stick thing quite well. It's not going to be such a stretch for them to learn the carrot trick too.
We import our math and science talent and if they stay, fine. If they don't, they take that valuable talent back to their own countries. What should be the focus here is that it is their countries, those other countries, who educated and trained them. The longer immigrants stay in this country, the more likely they are to forget that training when it comes to educating their own children. It is the relatively fresh citizens who immigrated from countries that have higher standards in education who succeed with their work and study habits and yes, the educations that were given to them in their native countries.
If you think that Japan lacks creativity and inititive in business, look at the cars on the highway. Toyota ring a bell?

Posted by: KathyWi | December 11, 2008 10:10 AM | Report abuse

It seems like your article puts all the emphasis on what's good for the country, the economy, etc. Those are important issues, but are they everything?

What about the students' minds? Don't all of them count equally? Doesn't the average tell us anything?

I know you are not for sacrificing the many for the elite few, but your words like:

"What is likely much more important is how many high performers you have," and:

"...test scores, at least average test scores, don't seem to be related to anything important to a national economy," send up red flags.

If math ed and schools are only about what is "supply side," then the problem is bigger, and worse, than we think.

I think that is a horrible way to think about something that rather might be about helping humans live up to their potential as humans first, and as little consumers and producers maybe somewhere further down the line (imagine that!)

Brian (a.k.a. Professor Homunculus at )

Posted by: mathmojo | December 12, 2008 6:55 PM | Report abuse

The comments to this entry are closed.

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company