Network News

X My Profile
View More Activity


Posted at 2:18 PM ET, 02/ 7/2011

New study: How L.A. Times teachers data is flawed

By Valerie Strauss

Nobody should be surprised about a new study that finds big flaws with last year’s Los Angeles Times project in which it used “value added” methods to rate the effectiveness of more than 6,000 teachers.

But feel free to be annoyed: not at the results of the study’s findings, but rather that the people making important policy decisions – our education secretary, legislators, governors – keep ignoring experts who warn that such evaluation methods are invalid and unreliable.

The Times, for those who don’t know, published a statistical analysis of student test data last August to rate teachers in the Los Angeles Unified School District elementary schools. Its analysis, conducted by Rand Corp. senior economist Richard Buddin, used a form of measurement that uses student test data and other factors about teachers to evaluate how effective they are it. They are called “value-added” methods, and they are all the rage in education reform.

When The Times published its report, experts who oppose using standardized test scores as a sole measure for evaluation attacked it. Teachers cried foul (one committed suicide.) The Times stood by its database then. It still is. In fact, the headline of its Sunday story said, "Separate study confirms many Los Angeles Times findings on teacher effectiveness." It's hard to know how they came up with that assessment of the new critical report.

What's worse is that The Times project is getting more praise from journalistic circles at the very time its results are being questioned by experts; it just won an award administered by the National Institute for Computer-Assisted Reporting (a joint program of Investigative Reporters and Editors and the Missouri School of Journalism) and the Knight Chair in Journalism at the Walter Cronkite School of Journalism and Mass Communication at Arizona State University. I guess the judges know a lot about data, but nothing about schools or reform.

The authors of the new study -- being released Tuesday -- evaluated whether the evidence Buddin presented supports the use of value-added methods to evaluate teachers. Derek Briggs and Ben Domingue of the University of Colorado at Boulder also tried to replicate his results through an independent re-analysis of the same data that he had used. Their findings, the report says, “raise serious questions about Buddin’s analysis and conclusions.”

When they put the data through an alternative value-added model that used a long history of a student’s test performance as well as peer influence and school-level factors, the results changed dramatically. In fact, for reading results:

*46.4 percent of teachers who were rated would retain the same effectiveness rating under both models.
*8.1 percent of those teachers identified as effective under the alternative model are identified as ineffective in the L.A. Times database
*12.6 percent of those identified as ineffective under the alternative model are identified as effective by the L.A. Times model.

For math results:
*60.8 percent of teachers would retain the same effectiveness rating
*1.4 percent of teachers identified as effective under the alternative model are identified as ineffective in the L.A. Times model
*2.7 percent would go from a rating of ineffective under the alternative model to effective under the L.A. Times model.

The authors had other interesting findings as well, including evidence that conflicted with Buddin’s finding that traditional teacher qualifications have no association with student outcomes.

But the fact that a similar value-added formula could come up with such different results using the same data should give anyone pause about putting a teacher's livelihood on the line with this evaluation method, or for that matter, a school's reputation or a student's academic performance.

This new study follows a series of others that show plainly that using standardized test scores to evaluate teachers is a bad idea. For those who then ask how else teachers can be evaluated, the fact is that there are a number of multi-pronged evaluation systems that have been used in school systems and that are effective. I've published a number of posts about better evaluation methods for teachers, schools and students.

(The new report is being published by the National Education Policy Center at the University of Colorado at Boulder, with partial funding from the Great Lakes Center for Education Research & Practice, which itself is partly funded by a teachers union.)

A Times editor [a previous version of this story said he was a reporter on story], David Lauter, told my colleague Nick Anderson that he welcomes the new report, saying: "Part of the whole point in our putting all this effort into this work was to spark a public debate about how best to evaluate teachers. The more that people get into this, the better."

Actually, public schools would be better off if nobody had gotten into this. It's a waste of money -- the Gates Foundation has given hundreds of millions of dollars to experiment with this kind of evaluation -- and it's a sad detour from real reform.

That was true when Ronald Reagan pushed for merit pay when he was president 30 years ago, and it remains true. We never seem to learn.

Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!

By Valerie Strauss  | February 7, 2011; 2:18 PM ET
Categories:  Research, Standardized Tests, Teacher assessment, Teachers  | Tags:  l.a. times, latimes, los angeles times, los angeles times teachers, rand corp., rating schools, rating teachers, teacher effectiveness, teacher evaluation, teacher ratings, teachers, teachers database, value added, value added methods, value added models  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Best college essay ever?
Next: How can we fairly assess teachers, students, schools?

Comments

Hi Valerie. I'm a little unclear on the results of this study. When you write that:

*46.4 percent of teachers who were rated would retain the same effectiveness rating under both models.
*8.1 percent of those teachers identified as effective under the alternative model are identified as ineffective in the L.A. Times database
*12.6 percent of those identified as ineffective under the alternative model are identified as effective by the L.A. Times model.

what do the authors of this reanalysis say about the other 32.9% in the LAT study?

Posted by: Busboom | February 7, 2011 3:27 PM | Report abuse

Sadly, you've twisted/selectively quoted some of the facts from the LAT story (as well as the WP story) so as to fit nicely into your ideological point of view on reform. Which makes it really hard to take what you write seriously.

Posted by: frankb1 | February 7, 2011 5:44 PM | Report abuse

Thanks, Valerie, for continuing to remind people of some of the dangers of non-educators pontificating, presenting questionable data and otherwise influencing the direction of education in the U.S.

There are a couple of other points to remind people of:

1. The leadership in education - principals,vice-principals, school boards and superintendents need to be under the same intensity of scrutiny as teachers are put under, if not more; principals and vice-principals after all, do the hiring and firing of staff as well as set discipline poicies and testing schedules. School boards and superintendents have tremendous influence in determining facilities budgets and the general direction of an area's educational goals.

2. It's interesting to consider that most of today's power players in the 'reform' efforts are primarily men with ZERO teaching or education credentials: Bill Gates, Joel Klein, Jeb Bush, Sec. Duncan, City Mayors....and the women they get to go to bat for them are either big mouths (Rhee) or entertainers (Oprah).

In 28 years of teaching, for every ineffective teacher I saw, there were at least 40 who were dedicated, student-oriented, there for the long haul, and worked for little recognition or monetary rewards.

Posted by: PLMichaelsArtist-at-Large | February 7, 2011 5:58 PM | Report abuse

Thanks for telling the truth, Valerie. I appreciate you more than you will ever know.

Posted by: jlp19 | February 7, 2011 6:51 PM | Report abuse

frankb1,

Just read the study. Let's be frank here, shall we, Frank? What exactly did Valerie twist or cherry pick facts?

I did notice that other news organizations have twisted or cherry picked how they would report on this study, but Valerie did a decent job summing up the basic ideas in Briggs and Dominigue's study (excluding her obvious commentary at the end, though).

Posted by: DHume1 | February 7, 2011 8:38 PM | Report abuse

frankb1,

BTW, I just read the LA Times piece. After reading the study, I found that it grossly inflates and distorts the study. Before you criticize Strauss for not doing her job, I suggest that look at the primary source.

Posted by: DHume1 | February 7, 2011 8:55 PM | Report abuse

The WP Nick Anderson piece as well as the LAT story on this are pretty informative, balanced and nuanced. What is written here is the ranting of an ideologue. How is that helpful?

Posted by: frankb1 | February 8, 2011 1:35 AM | Report abuse

So you're saying, without really saying, that you didn't read the Briggs and Dominigue study but relied on secondary sources? Did you ever go to college, Frank?


And BTW, you never really addressed my first question. You basically repeated yourself and added one adverb and three adjectives to appear as if you did. That's a rhetorical tautology, Frank. Post again when you finally relieve yourself of that constipation-like blockage.

Posted by: DHume1 | February 8, 2011 8:33 AM | Report abuse

With all the emphasis on measuring teachers by how much their students' test scores improve, what happens when a teacher gets a group of top-scoring students who can't improve? Or when they improve one year in one year? Are these teachers considered ineffective?

Posted by: sideswiththekids | February 8, 2011 9:03 AM | Report abuse

Valerie,

Thank you so much for your reasoned & well-informed voice in the struggles we are in today in public education. I appreciate your analysis of the LA Times' response to the findings of the NEPC re-analysis of the LA Unified teachers' test-score data that the Times published against the advice of the education research community & the protests of the LA teachers whose professional reputations they damaged by making these "effectiveness scores" public. News organizations should not be in the business of education research. They are not bound to the same ethical & scientific standards as legitimate education researchers, both individual & institutional. They hired Richard Buddin as a lone researcher & his work was never peer-reviewed through the usual mechanisms that we use to ensure reliability & validity of methods & findings. One of these issues is replicability. If the LA Times data had been subjected to the standards of "due diligence" & protection of human subjects that are part of research ethics, these data would have never been made public, especially with individual teacher's names attached. This was & is a huge disservice to public education & to the 6,000 LAUSD teachers who have no place to go to get their professional reputations back.

One thing that puzzles me about the NEPC study is the language that attempts to distinguish between "average" teachers & "effective" teachers. The phrase in parenthesis in their summary is "(teachers rated as effective who are really average)" on p. 2 of the summary. This should be examined carefully since the federal definition (I paraphrase) of an "effective" teacher is one who achieves one year of academic growth in the majority of his/her students for each year of academic instruction. Does any data base show that the vast majority of teachers do not meet this definition using any VAM model? We must be very wary of attempts to compare & classify teachers as to more or less effective when in fact very few teachers can be shown not to be "effective" under the normal & acceptable definition of teacher effectiveness. This will shine the light of reason on the motivations behind use of the value-added model in the first place.

Posted by: jkmora | February 8, 2011 1:07 PM | Report abuse

Thanks for your information. As a teacher in Tennessee, I have had to "live" with the "value added" method of evaluation for a few years now. Few people know that the formula was designed by a Univ. of TN statistician . And the education system of Tennessee has had to suffer from it ever since. Even before No Child Left Behind. The use of it has NOT changed or reformed our systems. And has been of little benefit. When will the "powers that be" understand that education is not a corporation, it is a SERVICE. It cannot be evaluated like an industry. And , no, standardized test results do not show what children are learning ; nor do they show how well teachers are teaching. They only show which students take tests well, and which teachers teach to the test well. I hope someone will do something to get the attention of the federal government, and state governments as well, concerning standardized test scores. nmoore2

Posted by: nmoore2 | February 8, 2011 2:07 PM | Report abuse

i'm no fan of the LAT publishing individual teacher ratings or this latest attempt to cover the story but let's not confuse publishing individual ratings publicly and using test scores internally for teacher evaluation or support or other constructive purposes -- in that sense i think valerie's post goes too far in damning any and everything related to linking test scores and kids to teachers.

Posted by: alexanderrusso | February 8, 2011 3:29 PM | Report abuse

Hey Frank,

Do you remember what you wrote before? "The WP Nick Anderson piece as well as the LAT story on this are pretty informative, balanced and nuanced."

Take a look at this: http://nepc.colorado.edu/files/FactSheet_0.pdf


Briggs' response to the Times article is truly informative, balanced, and nuanced. And this is why, Frank, you either do the basic work and read the primary source in question or suffer the fate of eternally being the noob who speaks through his clogged-up bottom end.

Posted by: DHume1 | February 10, 2011 10:25 AM | Report abuse

The LA Times rightly brushed off this critique as more of the same from union supporters. Briggs' response was a temper tantrum.

That fact is that LA public schools are really terrible, almost as bad as DC the public schools. The primary reason: too many tenured classroom teachers are incompetent & ineffective.

To quote LA Mayor Antonio Villaraigosa:
"When we fought to change the seniority-based layoff system that was disproportionately hurting our neediest students, the teachers union fought back.

When we fought to empower parents to turn around failing schools and bring in outside school operators with proven records of success, the teachers union fought back.

And now, while we try to measure teacher effectiveness in order to reward the best teachers and replace the tiny portion who aren't helping our kids learn, the teachers union fights back.

It's not easy for me to say this. I started out as an organizer for UTLA (United Teachers Los Angeles), and I don't have an anti-union bone in my body. The teachers unions aren't the biggest or the only problem facing our schools, but for many years now, they have been the most consistent, most powerful defenders of the unacceptable status quo."

Posted by: frankb1 | February 11, 2011 10:52 PM | Report abuse

One reason why the Times, and the Los Angeles community, might have dismissed Briggs' critique without a second thought; the Los Angeles school district has spent $500,000 per teacher ($3.5 million total) trying to fire just SEVEN of the districts 33,000 teachers for poor classroom performance.

The LA community knows full well the extreme lengths the teachers union, and their allies, will go to protect incompetent and ineffective teachers.

Posted by: frankb1 | February 12, 2011 9:53 PM | Report abuse

And the LA community isn't buying the cynical exploitation of Rigoberto Ruelas suicide. You would think that the uber-progressive LA Weekly would be sympathetic, but they're not.

​"The apparent suicide of Rigoberto Ruelas, an LAUSD teacher at Miramonte Elementary School in South Gate, has led to rampant speculation by Los Angeles TV and radio stations that Ruelas couldn't bear to live following an L.A. Times series that published the names and scores of 6,000 teachers -- and showed Ruelas to be a below-average teacher whose students' progress somewhat lagged other kids at the same school.

Today's broadcast media speculation, quick to blame the Times, seems based on thin evidence.

First, word so far does not indicate that any note was found. And whose theory are KNX and KCAL and other radio and TV stations so heavily relying upon? There's some unnamed family members. Then there's a South Gate Police Department spokesman named Tony Mendez, who told the local CBS affiliate KCAL, "He was stressed out because of the teacher ranking in the L.A. Times because it wasn't a direct reflection of his performance."He was an outstanding teacher," Mendez went on. "His coworkers have praised him for the outstanding work he does at work. His students always speak very highly of him.

If Officer Mendez is a psychiatrist, he's not identified as such by KCAL, and let's stipulate that he's not. Suicide is one of the most mysterious of all human mysteries.To pin the blame on what amounts to a bad performance review at work is quite a thing.

And it's not as if the L.A. Times or its series could have any direct impact on Ruelas...... as in any suicide, there are many questions unanswered: Did he suffer from a history depression? How was his family life? What kind of life did he lead outside work? Dr., er, Officer Mendez offers no clues to any of these questions. Although in addition to his vast knowledge of psychiatry, Mendez is apparently also an education expert and knows the teaching abilities of Ruelas.

Except that, you know, more than likely, this officer hasn't a clue what he's talking about. But he's a local cop, so, hey, that's good enough for local media.

Not surprisingly, the teachers union is exploiting this for all it's worth."

http://blogs.laweekly.com/informer/2010/09/rigoberto_ruelas_suicide_at_la.php

Posted by: frankb1 | February 13, 2011 7:24 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company