Jay Mathews (and Obama) vs. me on tests
“Valerie says the standardized tests we use now are too unreliable to tolerate.” -- Jay Mathews
Actually, I didn’t say that.
But my unrivaled colleague Jay Mathews wrote that I did, just before he went on in his Class Struggle blog, to extol the virtues of standardized tests. He also complained that I had not provided any evidence for “my side.”
So let’s start with exactly what my side is. I am not anti-test. What we should not do, but have done for the last eight years under No Child Left Behind, is use results of standardized tests to make major decisions about personnel and the fate of schools. On the basis of standardized test scores alone, for example, excellent schools were deemed to be failing because of NCLB's irrational accountability formula. President Obama, in his newly stated vision for rewriting NCLB, rightly said he wants to get rid of that scheme.
The results of standardized tests also should not, by themselves, determine whether a teacher gets merit pay, is fired or promoted, etc. This is what Obama and Education Secretary Arne Duncan, wrongly in my view, are advocating in part, and that is what I objected to in the post on Sunday to which Jay refers.
Opposition to using standardized tests for high-stakes decisions--including to measure how effective teachers are in the classroom--hardly originates with me. Experts in assessment have been complaining about it for most of the eight years that the country’s public schools have become obsessed with testing under NCLB.
The reason, they say, is that we don’t have standardized tests that are drawn well enough to be a single measure of anything important. The tests are just not that sophisticated, and, besides, all tests are not born equal.
Some state tests are more aligned with what kids are supposed to be learning than others. The federal National Assessment of Educational Progress--called "the nation's report card" because it is the only standardized test given in districts across the country--sometimes produces results that vary widely from state test scores.
So which tests should high-stakes decisions rest upon?
Daniel Koretz is a professor at Harvard University's School of Education and author of the book, “Measuring Up: What Educational Testing Really Tells Us.” He’s non-partisan when it comes to testing--that is, he’s not for or against it. He just researches them.
In the book, he explains in great detail the limits of standardized testing. No single test can tell us everything a child has learned or knows. He and others note that there is a certain amount of measurement error in every test. (And there are other complications as well; for example, when the tests are taken by a child with a disability that limits his or her chances of having the same conditions as other children. Some kids are given accommodations, but not many.)
Koretz also discusses how any single indicator used for high-stakes decision-making is more likely to become corrupted. Teaching to the test, which is what happened in many schools under NCLB, is one such way scores become corrupted.
Here’s what he said in an interview on this Web site:
“The misconception that matters the most is the notion somehow a good test measures all of what’s important. A good test is like a political poll. It’s a very small sample of something much larger.... So just as you predict a presidential election by polling 500 or 700 or 1,000 out of 120 million voters, you sample from this big domain of achievement a modest number of things that allow you to predict the whole. That’s all a test is, and its value is only as a tool for estimating what kids really know about the whole. Failing to understand that underlies, I think, a lot of the unfortunate consequences of high stakes testing today because people think that if they teach what’s on the test, they must be doing the right thing.
"It’s really a matter of degree, that if the pressure becomes too severe, then people game the system. And this is not a problem limited to education; it’s just everywhere you look. So for example, some years ago, the British National Health Service imposed time limits on the amount of time that patients could be waiting in emergency rooms, which for people who’ve waited in emergency rooms would seem like a very good thing to do. And it was a good thing to do, but unfortunately people gamed the system in a number of ways, one of which is that some hospitals kept patients in queues of ambulances out on the street until they had enough room that they were confident that they could get them through in four hours. Well, the answer to that problem isn’t, stop worrying about wait times. It’s [to] find a better way to hold hospitals accountable for keeping wait times short. And the same is true in education. The answer to the current problems we’re seeing is not, in my view, stop holding schools accountable for teaching kids. It’s [to] find a better way to do it, one that has fewer side effects."
That makes sense to me.
Now, Jay says that he has seen a lot of test results over the last 30 years that “seem to conform” with what he knows of the quality of teaching and the socio-economic level of the students being tested.
Research has long shown that students who live in high- and middle-income areas do better on tests--and in school--than students who live in low-income areas. Obviously the lucky kids are exposed to more experiences at a younger age, more words, better teachers and health care, schools with more resources, etc. etc.
Research that attempts to explain the variance in test scores across populations of diverse groups of students shows that family and demographic variables explain the largest part of total explained variance. Among commonly collected family characteristics, the strongest associations with test scores are parental educational levels, family income, and race and/or ethnicity. Secondary predictors are family size, the age of the mother at the child’s birth, and family mobility. Other variables, such as being in a single-parent family and having a working mother, are sometimes significant after controlling for other variables. The states differ significantly in the racial or ethnic composition of students and in the characteristics of the families of students, so it would be expected that a significant part of the differences in the NAEP test scores might be accounted for by these differences.
Jay also wrote that “schools that have taken unusual measures to deepen and invigorate the learning of impoverished children, such as Achievement First, Uncommon schools and KIPP, show significantly better scores than schools that have not.”
Well yes, but for one thing, that isn’t simply because the teacher in the classroom worked miracles all alone. It is because the teacher in the classroom was one of a number of measures, that, together, helped the student succeed.
How fair, then is it, for individual teachers to be judged strictly on the basis of standardized test results, without any of the other factors in a child’s life being changed?
If teachers get no support for what they do from within their school, if they work in a non-collegial atmosphere with few resources, how is it fair to judge them entirely on the scores of their students?
Why is the teacher being held completely responsible for student achievement when we know that many factors go into student performance?
It doesn’t make sense to me.
Tell me if/why I’m wrong and Jay Mathews is right/wrong.
Follow my blog all day, every day by bookmarking washingtonpost.com/answersheet And for admissions advice, college news and links to campus papers, please check out our new Higher Education page at washingtonpost.com/higher-ed Bookmark it!
| March 16, 2010; 1:15 PM ET
Categories: Education Secretary Duncan, No Child Left Behind, Standardized Tests | Tags: No Child Left Behind, standardized tes
Save & Share: Previous: Latest college trend: Mixed-gender dorm rooms
Next: Terps Coach Williams: Keep college advising to yourself
Posted by: Linda/RetiredTeacher | March 16, 2010 1:43 PM | Report abuse
Posted by: mamoore1 | March 16, 2010 1:48 PM | Report abuse
Posted by: Linda/RetiredTeacher | March 16, 2010 2:18 PM | Report abuse
Posted by: qaz1231 | March 16, 2010 4:20 PM | Report abuse
Posted by: aed3 | March 16, 2010 7:49 PM | Report abuse
Posted by: Trev1 | March 17, 2010 1:17 AM | Report abuse
Posted by: DavidBCohen | March 17, 2010 2:00 AM | Report abuse
Posted by: Trev1 | March 17, 2010 3:24 AM | Report abuse
Posted by: inthetrenches1 | March 17, 2010 9:05 AM | Report abuse
Posted by: Linda/RetiredTeacher | March 17, 2010 10:36 AM | Report abuse
Posted by: clevin | March 19, 2010 10:47 AM | Report abuse
The comments to this entry are closed.