LA Times reporters respond to my questions on school ratings
I have been avidly following the Los Angeles Times publication of value-added data from elementary schools in the L.A. district. After Sunday's latest installment, I posted an appreciation of what they were doing, but expressed frustration with some gaps in the story, and what I considered at least slight misrepresentation of some of their data. I thought they had failed in particular to address the possible statistical distortions that arise when analyzing how well already high-performing schools do in raising student achievement higher than what those students have done previously.
The reporters, Jason Felch, Jason Song and Doug Smith, responded Tuesday. Here is what they sent me:
Thanks for these good questions. As you suggested, some of the broader context you’re wanting will be available when the school and teacher database goes up later this week. Here are some answers for the interim:
Are the average losses by students at Wilbur – 10 percentile points in math and 3 in English over three years – meaningful? We said in our first story that teachers matter most, and have three times more impact on student scores than schools. So the lower school effect numbers should not come as a surprise. They may strike you as small, but they paint a dramatically different picture than the school’s API score, and we think they will be very meaningful to parents. Also, keep in mind that school effects, while small, are far more stable than teacher effects because they’re based on hundreds and hundreds of individual students.
You’re right to point out that students at Wilbur still end their time at the school as high achievers, even after sliding while there. But don’t we expect all children at schools to grow, not only those who are below proficient? Value-added reveals what raw achievement tests obscure: many high achieving kids at this and other schools are not being challenged. The gains of low-achieving students are often ignored.
You have several questions relating to the so-called “ceiling effect.” Are kids at the top of the achievement spectrum topping out the test, and therefore unable to make significant gains? Like other research in this area using California data, we found no evidence of that, as the story says. Presumably, if there were a ceiling effect you would see mostly low-achievement schools at the top of the growth list, while high-achievement schools would be pushed down. We found the opposite: many of the top schools in the district by growth also topped the lists of API scores. If anything, schools with lots of high achievers appear to have an easier time showing growth. (This may be because gifted kids have a very small but statistically significant advantage on growth scores, as you can see in our methodology paper.) We focused our report on those schools where the API was misleading, and might have made this point more clearly in the story. As you hinted, space limitations didn’t allow for that elaboration. But it will be very clear when parents start using the data on our database.
As for outside research on ceiling effects with California’s state test, see Cory Koedel & Julian Betts, 2009. "Value-Added to What? How a Ceiling in the Testing Instrument Influences Value-Added Estimation," NBER Working Papers 14778, National Bureau of Economic Research, Inc.
You ask about the precision of the data in several places. We’re working hard to make sure the information we’re going to make public is understood in the proper context, and with the appropriate caveats in place. For example, teachers and schools will not be receiving numbered percentile rankings, which imply a level of precision that is not possible with these estimates. Rather, their relative position will be shown along a spectrum of effectiveness in a way that visually indicates the inherent imprecision of these estimates. A Q&A will also inform parents how they might balance this information with other indicators of teacher and school effectiveness. For more technical readers, our methods paper (www.latimes.com/teachermethod) gives the actual effect sizes found in our study, expressed in standard deviations from the mean.
A few points we would like to make in addition to those you touched on:
There has been much debate about the reliability of value-added approaches, and there are ample ways to show it is a far from perfect measure. But shouldn’t we really be comparing it to the status quo for the vast majority of school districts – occasional, pre-announced and subjectively evaluated classroom visits, once every few years? And what of the reliability of some of the other “multiple measures” everyone agrees should be a component of teacher evaluations? What is the “error rate” of parent and student surveys, portfolio reviews, and the various observation rubrics? The answer: nobody knows. That’s why the Gates Foundation is spending millions to test them, using value-add as a baseline. When compared to these other largely unstudied measures of teacher effectiveness, experts tell us value-added vaults to the top of the class.
I would point your technical readers to this 2008 paper by Kane and Staiger using data from LAUSD: http://opr.princeton.edu/seminars/papers/kane_and_staiger3-30-2010.pdf
It is a random-assignment experimental validation of several value-added methodologies. The conclusion: “While all of the teacher effect estimates we considered were significant predictors of student achievement under random assignment, those that controlled for prior student test scores yielded unbiased predictions and those that further controlled for mean classroom characteristics yielded the best prediction accuracy.” Teacher effects, they also found, faded by 50 percent per year.
Finally, value-added measures are being adopted by school districts across the country, and usually in secrecy. They are complex, and depend greatly on a variety of decisions made along the way. Yet both the results and the methodology have often been kept away from teachers, parents, experts, and others. As you know, the failure of DC schools to make their technical methodology public before using the scores to fire teachers led to quite a ruckus in your town.
Given that history, we thought it important that the approach be publicly vetted and discussed, even if it meant fielding technical challenges for months to come. We’ve gone to lengths to be transparent about our process and decision making along the way. For fairness, for months we’ve had a team of web and data people building a back-end verification system that allows us to give teachers an opportunity to view and comment on their data before it is made public. We’ve made our methodology public and welcomed the scrutiny of experts across the country. And we’ve gone to lengths to explain those methods in lay terms to our readers.
Thanks for your interest in the stories.
Me again: They also said they would try to get to me some of the data I thought was needed on high performing schools that did well on their measures, compared to Wilbur Elementary, the school their latest story focused on that did not do so well. I will post that when I get it.
| August 25, 2010; 1:44 PM ET
Categories: Jay on the Web | Tags: LA Times response to Mathews questions, data base to be released later in week
Save & Share: Previous: Why it's okay to be bored in school
Next: Please Mr. President, don't speak on school time
Posted by: Linda/RetiredTeacher | August 25, 2010 2:50 PM | Report abuse
Posted by: Linda/RetiredTeacher | August 25, 2010 3:03 PM | Report abuse
Posted by: richardcmiller | August 25, 2010 5:21 PM | Report abuse
Posted by: johnt4853 | August 25, 2010 5:52 PM | Report abuse
Posted by: bsallamack | August 25, 2010 6:18 PM | Report abuse
Posted by: Jay Mathews | August 25, 2010 7:10 PM | Report abuse
Posted by: Cal_Lanier | August 25, 2010 9:49 PM | Report abuse
Posted by: cypherp | August 26, 2010 8:27 AM | Report abuse
Posted by: holland21 | August 26, 2010 9:04 AM | Report abuse
Posted by: mcstowy | August 26, 2010 10:19 AM | Report abuse
Posted by: johnt4853 | August 26, 2010 10:57 AM | Report abuse
Posted by: bk0512 | August 26, 2010 12:42 PM | Report abuse
Posted by: bsallamack | August 26, 2010 12:44 PM | Report abuse
Posted by: bsallamack | August 26, 2010 1:00 PM | Report abuse
Posted by: bsallamack | August 26, 2010 1:28 PM | Report abuse
Posted by: celestun100 | August 26, 2010 3:41 PM | Report abuse
Posted by: edlharris | August 26, 2010 4:18 PM | Report abuse
Posted by: bsallamack | August 26, 2010 5:00 PM | Report abuse
Posted by: bsallamack | August 26, 2010 5:18 PM | Report abuse
Posted by: sobronxschool1 | August 26, 2010 8:10 PM | Report abuse
Posted by: musiclady | August 26, 2010 8:31 PM | Report abuse
Posted by: celestun100 | August 26, 2010 9:54 PM | Report abuse
Posted by: alexanderrusso | August 27, 2010 12:31 AM | Report abuse
Posted by: bsallamack | August 27, 2010 2:16 AM | Report abuse
Posted by: bsallamack | August 27, 2010 2:21 AM | Report abuse
The comments to this entry are closed.