New study: How L.A. Times teachers data is flawed
Nobody should be surprised about a new study that finds big flaws with last year’s Los Angeles Times project in which it used “value added” methods to rate the effectiveness of more than 6,000 teachers.
But feel free to be annoyed: not at the results of the study’s findings, but rather that the people making important policy decisions – our education secretary, legislators, governors – keep ignoring experts who warn that such evaluation methods are invalid and unreliable.
The Times, for those who don’t know, published a statistical analysis of student test data last August to rate teachers in the Los Angeles Unified School District elementary schools. Its analysis, conducted by Rand Corp. senior economist Richard Buddin, used a form of measurement that uses student test data and other factors about teachers to evaluate how effective they are it. They are called “value-added” methods, and they are all the rage in education reform.
When The Times published its report, experts who oppose using standardized test scores as a sole measure for evaluation attacked it. Teachers cried foul (one committed suicide.) The Times stood by its database then. It still is. In fact, the headline of its Sunday story said, "Separate study confirms many Los Angeles Times findings on teacher effectiveness." It's hard to know how they came up with that assessment of the new critical report.
What's worse is that The Times project is getting more praise from journalistic circles at the very time its results are being questioned by experts; it just won an award administered by the National Institute for Computer-Assisted Reporting (a joint program of Investigative Reporters and Editors and the Missouri School of Journalism) and the Knight Chair in Journalism at the Walter Cronkite School of Journalism and Mass Communication at Arizona State University. I guess the judges know a lot about data, but nothing about schools or reform.
The authors of the new study -- being released Tuesday -- evaluated whether the evidence Buddin presented supports the use of value-added methods to evaluate teachers. Derek Briggs and Ben Domingue of the University of Colorado at Boulder also tried to replicate his results through an independent re-analysis of the same data that he had used. Their findings, the report says, “raise serious questions about Buddin’s analysis and conclusions.”
When they put the data through an alternative value-added model that used a long history of a student’s test performance as well as peer influence and school-level factors, the results changed dramatically. In fact, for reading results:
*46.4 percent of teachers who were rated would retain the same effectiveness rating under both models.
*8.1 percent of those teachers identified as effective under the alternative model are identified as ineffective in the L.A. Times database
*12.6 percent of those identified as ineffective under the alternative model are identified as effective by the L.A. Times model.
For math results:
*60.8 percent of teachers would retain the same effectiveness rating
*1.4 percent of teachers identified as effective under the alternative model are identified as ineffective in the L.A. Times model
*2.7 percent would go from a rating of ineffective under the alternative model to effective under the L.A. Times model.
The authors had other interesting findings as well, including evidence that conflicted with Buddin’s finding that traditional teacher qualifications have no association with student outcomes.
But the fact that a similar value-added formula could come up with such different results using the same data should give anyone pause about putting a teacher's livelihood on the line with this evaluation method, or for that matter, a school's reputation or a student's academic performance.
This new study follows a series of others that show plainly that using standardized test scores to evaluate teachers is a bad idea. For those who then ask how else teachers can be evaluated, the fact is that there are a number of multi-pronged evaluation systems that have been used in school systems and that are effective. I've published a number of posts about better evaluation methods for teachers, schools and students.
(The new report is being published by the National Education Policy Center at the University of Colorado at Boulder, with partial funding from the Great Lakes Center for Education Research & Practice, which itself is partly funded by a teachers union.)
A Times editor [a previous version of this story said he was a reporter on story], David Lauter, told my colleague Nick Anderson that he welcomes the new report, saying: "Part of the whole point in our putting all this effort into this work was to spark a public debate about how best to evaluate teachers. The more that people get into this, the better."
Actually, public schools would be better off if nobody had gotten into this. It's a waste of money -- the Gates Foundation has given hundreds of millions of dollars to experiment with this kind of evaluation -- and it's a sad detour from real reform.
That was true when Ronald Reagan pushed for merit pay when he was president 30 years ago, and it remains true. We never seem to learn.
Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!
| February 7, 2011; 2:18 PM ET
Categories: Research, Standardized Tests, Teacher assessment, Teachers | Tags: l.a. times, latimes, los angeles times, los angeles times teachers, rand corp., rating schools, rating teachers, teacher effectiveness, teacher evaluation, teacher ratings, teachers, teachers database, value added, value added methods, value added models
Save & Share: Previous: Best college essay ever?
Next: How can we fairly assess teachers, students, schools?
Posted by: Busboom | February 7, 2011 3:27 PM | Report abuse
Posted by: frankb1 | February 7, 2011 5:44 PM | Report abuse
Posted by: PLMichaelsArtist-at-Large | February 7, 2011 5:58 PM | Report abuse
Posted by: jlp19 | February 7, 2011 6:51 PM | Report abuse
Posted by: DHume1 | February 7, 2011 8:38 PM | Report abuse
Posted by: DHume1 | February 7, 2011 8:55 PM | Report abuse
Posted by: frankb1 | February 8, 2011 1:35 AM | Report abuse
Posted by: DHume1 | February 8, 2011 8:33 AM | Report abuse
Posted by: sideswiththekids | February 8, 2011 9:03 AM | Report abuse
Posted by: jkmora | February 8, 2011 1:07 PM | Report abuse
Posted by: nmoore2 | February 8, 2011 2:07 PM | Report abuse
Posted by: alexanderrusso | February 8, 2011 3:29 PM | Report abuse
Posted by: DHume1 | February 10, 2011 10:25 AM | Report abuse
Posted by: frankb1 | February 11, 2011 10:52 PM | Report abuse
Posted by: frankb1 | February 12, 2011 9:53 PM | Report abuse
Posted by: frankb1 | February 13, 2011 7:24 PM | Report abuse