Network News

X My Profile
View More Activity

Test that makes U.S. look bad may not be so good

Politicians and pundits are using results from the Programme for International Student Assessment|(PISA) tests to say our kids are falling behind the rest of the world, so maybe we should get some PISA practice. Brookings Institution scholar Tom Loveless, a member of the U.S. advisory board to PISA, offered this sample question for 15 year olds from the mathematics literacy section of the exam:

For a rock concert a rectangular field of size 100 m by 50 m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?
A. 2000
B. 5000
C. 20000
D. 50000
E. 100000

I think this is a bad question, and not just because I got it wrong. I said 5,000 The answer is 20,000. I don’t see why deciding four people, not one, would fit better in a square meter is a sign of math literacy. There are some people I don't like to get close to at concerts.

Loveless, an expert on international testing, agrees that the problem was ill-chosen. “I think it would throw kids off,” he said. “The math is rather trivial.” There are other problems with PISA, such as an ideological bias and a tendency to assume cause-and-effect relationships. But the American results on those exams — we are below average in both math and science literacy — are often cited as a national disgrace.

A key failing of PISA, Loveless said, is “it does not measure what kids have learned in school.” Why not? Because PISA exams are written by the losing side in a century-old debate over how to teach math. For convenience, call the pro-PISA people progressives and the anti-PISA people traditionalists. Loveless, who taught sixth grade in California for nine years before getting his Ph.D. in education from the University of Chicago, is a traditionalist but appreciates the arguments on both sides. The progressives want to make math instruction more relevant to the real world, and emphasize mathematical reasoning more than calculation. The traditionalists say you can’t reason well without mastering the fundamentals. They dislike their approach being dismissed by progressives as “shopkeeper math,” Loveless said, “like it was old-fashioned to try to compute anything.”

Loveless prefers the other major international test, Trends in International Mathematics and Science Study, which aligns with the way U.S. students are taught. He is the U.S. representative to the General Assembly of the International Association for the Evaluation of Educational Assessment, a 60-nation body that oversees TIMSS.

Policy makers are right to worry about our world standing in math and reading. But their interpretations of the data are often flawed, and make our schools look worse than they are. The Wall Street Journal headlined a 2004 PISA report: “U.S. Teens Are Among Worst at Math.” But on TIMSS we are above the international average in both 4th grade and 8th grade math.

Many PISA questions are refreshingly challenging, but conservative politicians who praise the test's rigors should be careful, since the attitudinal parts of PISA lean left. On PISA’s student questionnaire, those who support statements such as “I am in favor of having laws that regulate factory emissions even if this would increase the price of products” are deemed to be environmentally responsible. Those who disagree are not.

This is not to say international comparisons are useless. We can learn from the differences between us and the countries at the top of the TIMSS (and in some cases also the PISA) list —Hong Kong, Korea, Singapore, Taiwan and Japan. Their families tend to devote more out-of-school time to learning. Their curricula and textbooks are leaner and more focused. Their teachers often train in academic disciplines, not at education schools.

In those countries, hard-working students are admired by their peers. Here they are often dismissed as freakish nerds. “America has always been ambivalent about how academics fit in our culture,” Loveless said.

That’s okay with me. I think our economy, politics and society have benefited from having as much respect for street smarts as book smarts. Being unimpressed with academic degrees saves us from the stultifying aspects of European-style class divisions. But we pay a price. Perhaps it would help if we found ways to celebrate our grade school math whizzes as much as our playground stars. We want to use the right tests, but more hoorahs, even from 15 year olds, for people who do their homework can’t hurt.

By Jay Mathews  | October 19, 2009; 6:00 AM ET
Categories:  Metro Monday  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: Solve These International Test Math Problems
Next: Making the grade not about race, but culture


Jay, of course knowing the answer is 20,000 and not 5,000 is a sign of math literacy. It would be incredibly wrong to add the variable, "how comfortable are you around people." I am amazed that you are so blithly dismissing the question simply because you got it wrong due to you adding your fear of crowds into the equation (and really, no equation is necessary, common sense, attending concerts/football games, could have told you the best estimate).

In addition, your comments at the end of the article, that it is okay with you that American students dismiss hardworking students as "freakish nerds" goes completely against your views that all students should take AP courses (unless you are saying AP should be dumbed down in order to prevent only "freakish nerds" from taking those courses).

Posted by: researcher2 | October 19, 2009 7:04 AM | Report abuse

I have to agree that the rock concert question is off base. I got it right, but only because I've been to a few rock concerts since I was 15 and know that people stand cheek-by-jowl. When I was 15, however, I had been to wholly different kinds of outdoor concerts, where 2,000 in a 5,000-meter space would have been about right. (Think picnic blankets and coolers.)

Stanford mathematician James MIlgram left a long critique of PISA in a comment on our website about a year ago after we published an interview with PISA head Andreas Schleicher. Milgram quoted Finnish educators who criticized their students' mathematics abilities and questioned PISA's credibility in math. Milgram is, of course, decidedly conservative.

PISA offers the most dramatic data for those who stress the urgency of school reform. It serves political rather than curricular goals. Few people who use PISA in their advocacy go so far as to advocate for the vision of mathematics PISA reflects--or for any specific vision of mathematics in schools, for that matter.

Still, we don't necessarily need PISA to know that school improvement is an urgent matter.

Posted by: ClausvonZastrow | October 19, 2009 8:16 AM | Report abuse

A question that requires a subjective perception in a calculation with a right-or-wrong answer is a violation of the basic premise of empirical science. It allows, as demonstrated, someone to say you are wrong if your common sense does not coincide with mine.

It is further an error of statistical inference. If real scientists were posed with this question, the protocol would require that they explain their basis for estimating crowd density. And that is real nerdy stuff.

Posted by: dlnewquist | October 19, 2009 12:25 PM | Report abuse

My father spent close to 40 years in education. He taught, became one of the first master teachers in the New England states, moved up to become the department chairman, then principal of my high school and finally the superintendent of schools (and I survived the experience!). Later he became the Governor of NH's chairman for the Committee of Curriculum Review and the state representative to the National Council of Mathematics Teachers as they produced the first look at math standards at the primary and secondary school level. I think they were called Math2000 or 2005, something like that. The booklet was pretty colorful, the standards pretty rigorous. I saw this in 1980 or so. My father was pretty frustrated and discouraged by the fact that everyone wanted to go via the PISA route and not TIMSS. TIMSS places us in a much better light with regard to comparisons between the US and the rest of the world, but it also reflects a difference in philosophy between how we train teachers and develop curriculum in the US and how the other high ranking countries in the world do it. Essentially my father hated education courses as "moronic and useless". He felt everything should come out of an academic major, as in Mathematics, or Physics, or English Language Arts. This gives the teacher a thorough grounding in the subject matter and the teaching aspects can be taught with practice and mentoring. After teaching in and out of the military for some 30 years, I have come to agree with him.

Posted by: flyfisher_20750 | October 19, 2009 12:32 PM | Report abuse

The question was perfectly good. The real issue is whether you know the difference between a meter and a foot (factor of 3) and a square meter and a square foot (factor of 9). You would only need a square meter to stand in if you were 5 ft 4 inches tall and weighed 800 pounds. I would be more comfortable with only three people in my square meter, but four could fit.

Posted by: sscritic | October 19, 2009 12:38 PM | Report abuse

Since someone asked what would happen if the question were posed to a "real scientist", so I thought I'd answer. [PhD Physics].

100*50 is fairly easy, 5000 sq.m.

Then I thought about a square meter, a little more than 3 ft by 3 ft. My first guess was about 9 folks could be squished in there, so I gave it a rough rounding of 10 for about 50,000. That's on the high end, so I might back off to around 20,000.

The real question is how many people can fit in one square meter. It's a spatial question with some trivial math, not a math question.


Posted by: FairlingtonBlade | October 19, 2009 1:11 PM | Report abuse

sscritic said:

"The real issue is whether you know the difference between a ... square meter and a square foot (factor of 9)."

Then it looks like you'd get the question wrong :P A square meter is larger than a square foot by approximately a factor of 10.76

Posted by: mytwocents | October 19, 2009 1:36 PM | Report abuse

To: flyfisher_20750

I can agree with your father for secondary education, but not primary. If teachers are trained in a particular subject, it should be because they are working with students who are ready to master the subject beyond the basics taught in the primary grades. A teacher's ability to thematically teach various subjects has been found to be helpful for bilingual and special ed students. Some students need to hear subjects presented in a way that is familiar to them, and cross-discipline or thematic units do just that.

Nevertheless, the sample question presented by Mr. Mathews is indeed a bad question for many reasons. There is an art to writing questions for standardized tests and some approaches simply don't work when presented to a diverse student audience. Organizations should be reasonable when scoring these questions and statistically determine if the question should be removed from the final, overall score whenever it appears that enough otherwise intelligent students get the question wrong.

Posted by: doglover6 | October 19, 2009 1:47 PM | Report abuse

"I think our economy, politics and society have benefited from having as much respect for street smarts as book smarts. Being unimpressed with academic degrees saves us from the stultifying aspects of European-style class divisions."

Jay, Class divisions in the U.S. may not be European-style but they exist just the same with their own coating of hypocrisy.

Street smarts may be celebrated in popular culture, that self-made man up by the bootstraps hogwash, but U.S. employers are sorting hires by using college degrees as the sieve, whether or not academics is germane to the job.

Street smarts alone are not going to provide economic advancement for members of the lower classes. The members of the upper classes make sure their kids, nerdy or not, learn enough PISA/TIMMS/whatever math to get into college and get their entrance ticket to the better paying jobs.

If anything the last couple decades have seen the upper classes intensifying their self-perpetuation through education reform's standardization and high stakes testing schemes, abetted by industry's whining over lack of skilled workers.

Street smarts these days means getting a degree, just ask any immigrant at the bottom of the social ladder.

Posted by: speakuplouder | October 19, 2009 1:56 PM | Report abuse

I agree the question isn't a math question, it is a question about estimation of units. Also darn useful.

Here's a fun question: Say I have a basement in house that is 18' by 35' The basement has 8' tall ceilings. If stacked neatly full with quarters ($0.25) how much money is likely in it?

Again, it's not a math question it's a matter of units and estimation. Those are very very important skills to an engineer (I teach engineering) and frankly our students are at best moderate at it. People don't back up and look at the question in context. They just do math. The idea of thinking about the math first is really important.

Posted by: bobtom222 | October 19, 2009 3:18 PM | Report abuse

That question seems to have cultural bias against cultures that prefer a greater amount of "personal space" like the predominant Anglo-Saxon U.S. norm. I probably would've guessed the correct answer since 5000 would seem "too obvious". But I agree that it's a bad test of math skills because it relies on an arbitrary assumption about the number of people that could fit into a square meter.

Posted by: CrimsonWife | October 19, 2009 4:07 PM | Report abuse

The quarter problem wouldn't be too bad, because quarters have a fixed width, while the concert problem is horrible, since is has to do with people's cultural perception of personal space (as well as fire code regulations in countries).

From my (admittedly brief research) it appears that American concert venues allow for about 0.5 square meters per person, so the answer in America should be 10,000...

Posted by: someguy100 | October 20, 2009 9:40 AM | Report abuse

Speakuplouder makes this good point, with which many would agree:

"Street smarts these days means getting a degree, just ask any immigrant at the bottom of the social ladder."

I am not sure Bill Gates, Steve Jobs, or several other of our most creative business leaders who don't have college degrees would buy this notion.

Posted by: Jay Mathews | October 20, 2009 2:06 PM | Report abuse

I'm with Loveless:
PISA isn't much of a math test. However, even though it isn't much of a math test, it is still a national embarrassment to do so poorly on it. And Jay, get serious. How many non-college grads do you think Gates and Jobs hire every year? Do you really think you can get either one of them to disagree with the statement "street smarts these days means getting a degree"? It's like suggesting that winning the lottery is a better career plan than college!

Posted by: wswilson1 | October 20, 2009 9:23 PM | Report abuse

Jay -- For another look at the differences between TIMSS and PISA and the dangers and costs of using these international tests to generate policy recommendations, see former National Center of Education Statistics Commissioner Mark Schneider’s piece on PISA (you can also check out OECD’s Andreas Schliecher’s subsequent testy exchange with Schneider).

Also, while It is true that the average PISA 2003 math literacy score of students in Finland, the highest performing OECD country, was 74 points better than the average score of students in the US., the real tragedy may be that across many OECD countries, including Finland, students from families with higher socio-economic status performed on average 40-66 points better on that test than than those from families with lower SES (depending on which factor you look at: parent occupational status, parent education, or # of books in the home).

Rather than trying to figure out how to outperform each other, maybe we should be trying to work together to to address this fundamental educational disadvantage for students from families with lower socio-economic status.

Posted by: TCHatch | October 22, 2009 5:47 AM | Report abuse

The comments to this entry are closed.

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company