Network News

X My Profile
View More Activity


Posted at 2:45 PM ET, 01/28/2011

Do tests really help students learn -- or was a new study misreported? -- Kohn

By Valerie Strauss

This was written by Alfie Kohn, the author of 12 books about education and human behavior. His latest, the forthcoming "Feel-Bad Education . . . And Other Contrarian Essays on Children & Schooling," will be published this spring by Beacon Press. He lives (actually) in the Boston area and (virtually) at www.alfiekohn. He blogs at The Huffington Post.

By Alfie Kohn
The relationship between educational policies and educational research is both fascinating and disturbing. Sometimes policy makers, including those who piously invoke the idea of "data-driven" practice, pursue initiatives they favor regardless of the fact that no empirical support for them exists (e.g., high-stakes testing) or even when the research suggests the policy in question is counterproductive (e.g., forcing struggling students to repeat a grade).

Sometimes insufficient attention is paid to the limits of what a study has actually found, such as when a certain practice is said to have been proved "effective," even though that turns out to mean only that it's associated with higher scores on bad tests.

Sometimes research is cited in ways that are disingenuous because anyone who takes the time to track down those studies finds that they actually offer little or no support for the claims in question. (Elsewhere, I've offered examples of this phenomenon in the context of assertions about the supposed benefits of homework -- along with details about some of the other ways in which research is under-, over-, or mis-used.)

Then there's the question of what happens when the press gets involved. It's no secret that the reporting of research is often, shall we say, disappointing: A single experiment's results may be overstated, or a broad conclusion vaguely attributed to what "studies show" despite the fact that multiple qualifications are warranted. Possible explanations aren't hard to adduce: tight deadlines, lack of expertise, or a reporter's hunger for more column-inches or prominent placement (hint: "the results are mixed at best " is not a sentence that advances journalistic careers).

Whether ideology may also play a role -- a tendency to play up certain results more than others -- is hard to prove. But last week I found myself wondering whether The New York Times would have prominently featured a study, had there been one, showing that taking tests is basically a waste of time for students.

After all, the Times, like just about every other mainstream media outlet, has been celebrating test-based "school reform" for some time now -- and, in its news coverage of education, routinely refers to "achievement," teacher "effectiveness," exemplary school "performance," and positive "results" when all that's really meant is higher scores on standardized tests. The media have a lot invested in the idea that testing students is useful and meaningful.

So we probably shouldn't have been surprised to discover that the Times ran a lengthy (30-something-inch) story on the second page of its national news section last week under the headline "Take a Test to Really Learn, Research Suggests." And it should be equally unsurprising that the study on which the story was based didn't really support that conclusion at all.

(I'm picking on the Times because of its prominence, but many other news organizations also featured this article and described the study in similar terms. Other headlines included: "Taking a Test Helps Learning More Than Studying, Report Shows," "Learning Science Better the Old-Fashioned Way," and "Beyond Rote Learning.")

We should begin by noticing that the study itself, which was published on-line in the January 20 issue of Science, had nothing to do with -- and therefore offered not the slightest support for -- standardized tests. Moreover, its subjects were undergraduates, so there's no way of knowing whether any of its findings would apply to students in K-12 schools.

The real problem with the news coverage, though, is twofold: On closer inspection there are issues with how both the independent variable ("Take a Test") and the dependent variable ("Really Learn") are described.

What interested the two Purdue University researchers, Jeffrey D. Karpicke and Jannell R. Blunt, was the idea that trying to remember something one has been taught can aid learning at least as much as the earlier process of encoding or storing that information. Their study consisted of two experiments in which college students either practiced retrieving information they'd learned or engaged in other forms of studying. The former proved more effective.

The type of retrieval practice used in the study was an exercise in which students recalled "as much of the information as they could on a free recall test." But the idea of retrieval practice need not involve testing at all. In an e-mail message, Karpicke told me, "The NY Times article emphasized 'testing,' which is unfortunate, because that's really irrelevant to our central point. . . .Students could engage in active retrieval of knowledge in a whole variety of ways that aren't 'testing,' per se." For example (as he explained in a subsequent message), they might put the book aside to see how much of it they can recall, try to answer questions about it, or just talk about the topic with someone.

In other words, the experiments didn't show -- and never attempted to show -- that taking a test works better than studying. They were really comparing one form of studying to another.

Then there's the question of outcome. When I said a moment ago that the study showed retrieval practice was more "effective," the most appropriate response would have been to ask what that word meant in this particular context: More effective at what?

In the first experiment, students were asked both verbatim questions and inference questions that drew on concepts in the text they had been given. In the second experiment, they either took a short-answer test of the material or were asked to create concept maps of that material from memory.

The researchers seemed impressed that practice retrieving facts worked better than making concept maps (with the text in front of them) at preparing students for a closed-book test even when the test itself involved making concept maps. But the students were tested mostly on their ability to recall the material, so it may not be surprising that recall practice proved more useful.

I would argue that this result says less about how impressive the method was than about unimpressive the goal was. Karpicke and Blunt weren't investigating whether students could construct meaning, apply or generalize concepts to new domains, solve ill-defined problems, draw novel connections or distinctions, or do anything else that could be called creative or higher-order thinking.

Now if testing -- or any other form of retrieval practice -- were shown to enhance those capabilities, that would certainly deserve prominent media attention. But this study showed nothing of the sort. Indeed, I know of no reason to believe that tests have any useful role to play in the promotion of truly meaningful learning.

The main contribution of the articles that were published about this study is to remind us of the importance of reading the actual studies being described. To understand why the description of this one was misleading, try to imagine a newspaper running a more accurate account -- one with a headline such as "Practice Recalling Facts Helps Students Recall Facts."

-0-

Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!

By Valerie Strauss  | January 28, 2011; 2:45 PM ET
Categories:  Alfie Kohn, Research, Standardized Tests  | Tags:  alfie kohn, high-stakes testing, new york times, rote learning, standardized testing, student on testing, testing, testing study  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: On this we should agree
Next: Study: Most high school biology teachers don’t endorse evolution

Comments

"I would argue that this result says less about how impressive the method was than about unimpressive the goal was. Karpicke and Blunt weren't investigating whether students could construct meaning, apply or generalize concepts to new domains, solve ill-defined problems, draw novel connections or distinctions, or do anything else that could be called creative or higher-order thinking."

Ummm, Alfie? Every single one of those so-called skills you mentioned (construct meaning, apply concepts to new domains, higher level thinking, etc.) requires students to KNOW stuff first. Higher-order thinking skills aren't transferable if you don't know anything about the topic you're applying the skills to.

So, isn't it AWFULLY impressive if we can find ways to help students more efficiently retain knowledge? This is a means to the end you so desire.

Posted by: AJGuzzaldo | January 28, 2011 4:43 PM | Report abuse

Tests have no useful role to play in the promotion of truly meaningful learning?

Posted by: frankb1 | January 29, 2011 12:35 AM | Report abuse

This is just another example of Alfie making his living as sideshow freak.

Posted by: physicsteacher | January 29, 2011 6:38 AM | Report abuse

"...regardless of the fact that no empirical support for them exists (e.g., high-stakes testing)..."

Mr. Kohn's count is a bit short. I've been working on a review of the research literature on the effects of testing. To this point, I have reviewed a few thousand studies, several hundred of which are included in two meta-analyses--of quantitative and survey studies--and a research summary of the qualitative studies. The text for this study is currently under review by a scholarly journal.

I've posted the quantitative studies source list and effect sizes here:
http://www.npe.ednews.org/Review/Resources/QuantitativeList.htm

...and hope to post the same for the survey and qualitative studies soon.

One of the best information sources I encountered in my work is a review in Perspectives of Psychological Science (vol.1, no.3, 2006), by Roddy Roediger of Washington University (St. Louis) and Jeff Karpicke, written while Karpicke was a graduate student there. It covers the literature on memory retrieval research back a hundred years. Few outside the field of memory & cognition seem to realize any longer that experimental research on the effects of testing on achievement was vibrant in the early part of the twentieth century, and fairly conclusive. Some of that research was extraordinarily well done, as is the more recent work by Roediger and his colleagues.

Contrast the careful work of these research psychologists with the all too common statements of prominent researchers in education and economics in just the past several years who glibly claim that there had been no research (either ever or prior to theirs) on the effects of testing on achievement.

Posted by: richardpphelps | January 29, 2011 8:26 AM | Report abuse

AJGuzzaldo is, of course, correct. It is imperative for students to KNOW something first in order to then be able to use it. This isn't exactly an earth shattering revelation but surprisingly, many will read this piece and nod their agreement with the author.

Alfie Kohn's convoluted reasoning, remarkably, pays his bills and keeps him living fairly high off the hog. Belmont, Massachusetts is not like living at W125 Street and Broadway in Harlem.

How he retains any kind of following is mind boggling, but somehow he manages to do so.

Posted by: phoss1 | January 29, 2011 8:32 AM | Report abuse

If you've spent a lot of money paying for education school, Alfie makes you feel (if you're not too bright to begin with) that it didn't all go to waste.

Posted by: physicsteacher | January 29, 2011 9:35 AM | Report abuse

I think the idea of taking practice quizzes on material you are in the process of learning can help you to focus. That is a practice that has been used for awhile. Testing without learning and practicing first is obviously not a very efficient method for learning.

Posted by: ubblybubbly | January 29, 2011 10:13 AM | Report abuse

You must understand that Alfie's teaching experience is teaching existentialism. In a prep school. Does this sound in the least bit typical? In the grand scheme of things, does anyone care whether our youth are well versed in existentialism? Are there right or wrong answers in existentialism? Does his experience qualify him to comment on all subjects and age groups, as he continues to pretend?

I don't why Valerie keeps giving an audience.

Posted by: physicsteacher | January 29, 2011 11:25 AM | Report abuse

Flash cards are good example of this effect. Trying to remember something, peeking at the answer if you can't, putting it back in the pile of ones to try again. As much as deluded people like Ms. Strauss want to deny it, there is a place for memorization in learning. Trying to learn a foreign language, for example, more or less requires a huge amount of it.
http://askakorean.blogspot.com/2010/01/koreans-english-acquisition-and-best.html

Alfie Kohn and these other loopy "educators" need to kept on the sidelines.

Posted by: staticvars | January 29, 2011 12:06 PM | Report abuse

I find it fascinating that whenever Ms. Strauss provides us a post written by Alfie Kohn, we get a cacophony of angry responses.

Never mind that he eloquently points out how mass media (and policy mavens) regularly distort the findings of fairly straight-forward research, and regularly mislead readers about what was actually studied and what was actually suggested by the study.

That he adds his personal take on standardized testing, after pointing out the study itself was not even about standardized testing, nor even about students in K-12, and we have a bunch of true believers with their panties in a twist.

Posted by: Incidentally | January 29, 2011 12:43 PM | Report abuse

For those thinking that tests have no place in education, what is balancing a checkboook? What would you consider completing an application for a job or a resume? What would you consider providing the exact change at the counter? How would you know and employee could mix the right stuff to make a recipe?

They are ALL tests!

Posted by: jbeeler | January 29, 2011 2:36 PM | Report abuse

@Incidentally

So we're supposed to let Alfie slide? He himself misrepresents his experience and his wisdom. So if he claims that we shouldn't assign homework, I shouldn't assign any, because his highness, veteran of existentialism class, said so?

I've already had to deal with the effects of dopes like him posting some tripe on the internet, having some administrator read it, and make it the law of the land.

If someday some nurse sticks a needle in your arm with the wrong dose because this person can't deal with quantities correctly, you can thank fools like Alfie for sticking his nose where it doesn't belong. You can thank yourself as well.

Posted by: physicsteacher | January 29, 2011 3:20 PM | Report abuse

So physicsteacher, are we supposed to then disregard the fact that media outlets misrepresent the findings or more studies than I care to count because you do not like the source of the argument? It often boggles my mind what passes for "good research" according to the media. Personally I always take everything I read with a grain of salt, wonder at their motives and dissect their methodology. However, the general public doesn't do that. They say to themselves, "Oh it was in NYT so it must be true." I think we need our media sources to be a little more responsible when reporting "scientific findings."

Posted by: teachermomnj | January 30, 2011 9:05 AM | Report abuse

So physicsteacher, are we supposed to then disregard the fact that media outlets misrepresent the findings or more studies than I care to count because you do not like the source of the argument? It often boggles my mind what passes for "good research" according to the media. Personally I always take everything I read with a grain of salt, wonder at their motives and dissect their methodology. However, the general public doesn't do that. They say to themselves, "Oh it was in NYT so it must be true." I think we need our media sources to be a little more responsible when reporting "scientific findings."

Posted by: teachermomnj | January 30, 2011 9:06 AM | Report abuse

"So physicsteacher, are we supposed to then disregard the fact that media outlets misrepresent the findings or more studies than I care to count because you do not like the source of the argument"

If so, then Alfie has a number of targets to choose from, but there's seems to be a theme for his choices. It's always against standardization, homework, grades, rigor, the ability of Americans to compete on a global scale and to PAY FOR ALL THE STUFF we've been putting on our credit cards, and now tests, which are often the only real chance our kids have of PRACTICING essential skills.

A couple of years ago some Kohn-clone wrote some article on the net claiming we shouldn't grade homework so as not to penalize kids "for trying". It became the official policy at our school so now you can't penalize kids for NOT TRYING. Kids now can hand in blank homeworks crying that they tried but it was too hard (because it took more than 15 seconds of thought). Thankfully, they get to think pretty intensely on tests, and in preparation for them.

If Kohn and his minions get there way, your kid will graduate from school without having to work at anything.

I wouldn't have a problem with Kohn if he confined his "rules" to the teaching of existentialism in prep schools. But he doesn't.

Posted by: physicsteacher | January 30, 2011 10:24 AM | Report abuse

I stumbled upon this article and am truly puzzled by the snarky commentary. Studying and testing are not the same thing. Studying is a process of acquiring mastery over material and is not rewarded in school except through testing, which presumably has been designed in ways that measure the results of the studying. If the NYT or anyone else misrepresented a study that looked at self-testing as a form of studying and instead uses it to promote testing, why does that need to result in an attack on the person who reported it?

I think the answer to my question is that we all are quietly aware of how fundamentally flawed our educational system is at this point. We may have ideas about how to fix the system, but we aren't empowered to make changes, and we're hamstrung by a fear that we might "lose" a generation if we make drastic changes that aren't perfect, so we quibble about the margins and grump about philosophers who dare to question orthodoxy. If just half this negativity were put into helping identify the problems in the "system" so they could be fixed, that energy would almost certainly jumpstart our kids up the ranks from near the bottom of the developed world.

Posted by: HSerof3 | February 4, 2011 10:28 PM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company