Accountability in DCPS: Details from teacher's IMPACT report
My guest is Aaron Pallas, professor of sociology and education at Teachers College, Columbia University. He wrote a piece last week about the IMPACT teacher evaluation in D.C. public schools, and this is a follow-up.
Pallas writes the Sociological Eye on Education blog for The Hechinger Report, a nonprofit, non-partisan education-news outlet affiliated with the Hechinger Institute on Education and the Media. Pallas has also taught at Johns Hopkins University, Michigan State University, and Northwestern University, and served as a statistician at the National Center for Education Statistics in the U.S. Department of Education.
I had argued that school districts such as those in Washington D.C. and New York City, which are usin “value-added” measures for high-stakes personnel decisions (such as deciding which teachers to grant tenure, lay off or fire), have an obligation to make the technical features of these measures available for public scrutiny.
“Value-added” measures attempt to isolate how much individual teachers are contributing to a student’s current achievement from other relevant factors, such as a student’s poverty status or her achievement in the preceding year. The goal is to determine whether students are learning more or less from a particular teacher than statistical models would predict they’d learn from a typical teacher, and then to base teacher evaluations in part on the results.
In many states and school districts, value-added measures are limited to teachers in grades four through eight, because all states currently test students in grades three through eight in reading and math, and two years of data are needed to estimate a teacher’s influence on student achievement.
I also noted that the procedure described in the DCPS IMPACT Guidebook for calculating a teacher’s value-added score, which involves subtracting students’ scores on the DC CAS from 2009 from their scores in 2010, was seriously flawed, because the scores for one grade are on a different scale from the scores for the adjacent grade.
I concluded that if the district were following the description it gave in the Guidebook, it had botched the calculation of value-added scores for teachers – and that these flawed calculations may have been used to justify firing 26 teachers and placing hundreds more at risk of being terminated next year.
Frederick M. Hess of the American Enterprise Institute, in a blog post entitled “Professor Pallas’s Inept, Irresponsible Attack on DCPS,” raised two major objections to my piece. The first was that my analysis rested on a document I found on the DCPS website rather than on phone calls or emails to DCPS. Second, Hess said I misrepresented the complex procedures that DCPS and its contractor, Mathematica Policy Research, used to calculate its value-added measures.
Hess and I both agree that “the simple subtraction exercise” I described in my first post wouldn’t result in accurate value-added scores. And that’s precisely my point: the procedures DCPS described in the Guidebook are seriously flawed.
But whereas Hess is willing to take on faith the validity of whatever DCPS and Mathematica actually did – simply because “there’s a growing industry that specializes in doing precisely this” – I am more skeptical, because the description provided in the DCPS IMPACT Guidebook is so off-base.
Below, I present evidence from a current teacher’s IMPACT report to show that what DCPS provided to teachers two weeks ago states that a simple subtraction exercise is used to calculate value-added scores.
I worked from the Guidebook because that is what is publicly available. It doesn’t take a leap of logic or faith to expect that the Guidebook should accurately describe how IMPACT works.
It is the only document that was made available to DCPS teachers to explain the new system by which they were to be evaluated – and possibly fired – in the most recent school year. I was also careful to qualify my statements – noting that “I cannot be sure that this is what happened” – for the simple reason that there’s no technical report.
Such a report would detail the methodology used to make the calculations, allowing outside experts to confirm or dispute IMPACT’s validity. It would also help teachers better understand their evaluations.
McGoldrick explained that the technical report for the IMPACT system is currently being finalized by Mathematica. “DCPS is considering releasing the document publicly once it is finalized,” she said.
Meanwhile, Mathematica has now posted a brief description on its website of the value-added procedures it developed for DCPS.
Stanford economist Eric Hanushek, who serves as a technical advisor to IMPACT, told me that IMPACT uses a fairly standard regression model, which predicts current-year achievement by taking into account prior achievement and a variety of student characteristics like age, gender and socioeconomic status.
This is reassuring, although I still have many questions about the details of the model, such as its reliance on a teacher’s performance solely in the 2009-2010 school year rather than multiple years. Hanushek also told me that he’s never looked at what DCPS reports about the value-added measures.
I’m happy to hear that DCPS seems not to have botched the calculation of the value-added scores by subtracting last year’s score from this year’s score in the calculation of actual or predicted student performance, even if this is what DCPS is telling teachers it’s doing.
And I still contend that teachers whose careers were placed in jeopardy by the results should have been notified of the methodology in advance, not after the fact. Also, I reserve the right to be critical of the procedures if and when they are made public. The weight of expert scholarly opinion is that many technical and practical issues must still be worked out before value-added measures can be fairly used in high-stakes personnel decisions.
So why has DCPS misrepresented the value-added methodology to teachers and the public?
I didn’t invent the procedures described in the Guidebook – it’s an official DCPS publication. Is it really plausible that these procedures were only intended to illustrate the logic of the value-added approach to a lay audience? Other districts, such as New York City, seem able to represent the actual methods used. Do DCPS officials believe that teachers don’t need to understand the system that’s being used to evaluate their performance, or, worse, that they aren’t capable of understanding it?
Here’s an example of how the misrepresentation in D.C. continues.
A teacher in IMPACT Group 1 – the group for whom 50 percent of the IMPACT score is based on Individual Value-Added (IVA) – generously provided me a copy of her actual IMPACT report for 2009-10. (Details have been changed to protect the teacher’s identity.) This teacher received an overall impact score that was in the “Effective” range. Pages 4-8 of her report make clear that the calculation of the value-added score is based on a measure of “growth” that involves subtracting a student’s score from 2009 from his or her score in 2010.
For example, in the section entitled “Your IVA Score for Reading,” Step 1 reads as follows: “We calculated the average reading growth of your students over the past school year. On the spring 2009 DC CAS (before they entered your class), your students’ average scale score was 443.8. On the spring 2010 DC CAS (after being in your class), your students’ average score was 547.2. Therefore, on average, your students grew 103.4 DC CAS points this past school year.”
Step 2 is: “We calculated the average reading growth of students like yours over the past school year. On the spring 2009 DC CAS, the average scale score of students like your 2009-10 students was 443.8. On the spring 2010 DC CAS, the average scale score of students like your 2009-10 students was 539.8. Therefore, on average, students like yours grew 96.0 DC CAS points.”
Finally, in Step 3: “We compared the average reading growth of your own students with the average reading growth of students like yours. Recall that your own students grew 103.4 DC CAS points and students like yours grew 96.0 DC CAS points. Thus, your students grew 7.4 DC CAS points more than similar students. Your raw value-added score, then, is +7.4.”
In its reporting to teachers, then, DCPS appears to be saying one thing – which is clearly incorrect – but doing another. The big question, then, is why?
McGoldrick of DCPS provided this explanation: “The full technical description of the value-added model designed for us by Mathematica Policy Research includes sophisticated statistical techniques that require specific expertise to understand fully.We wanted to make the model as accessible as possible to teachers so they could understand how they were being evaluated. Therefore, we focused our materials on conveying the core concepts of value-added to teachers, leaving the full description to our technical advisors who have specific expertise in value-added.”
Okay, but as we learned from the financial meltdown, there are serious risks in relying on technologies so complicated that hardly anyone – and perhaps no one setting policy for DCPS – truly understands them. Involving outside experts can diffuse responsibility so that no individual is solely in charge of decisions with real consequences.
Follow my blog all day, every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!
| August 5, 2010; 6:00 AM ET
Categories: D.C. Schools, Guest Bloggers, Research, Standardized Tests, Teachers | Tags: IMPACT and d.c., d.c. teacher evaluations, d.c. teachers, dc schools and teachers, how to evaluate teachers, value added evaluations
Save & Share: Previous: Ravitch: Mayoral control means zero accountability
Next: Principal: Not the change I had in mind
Posted by: adcteacher1 | August 5, 2010 7:53 AM | Report abuse
Posted by: axolotl | August 5, 2010 8:08 AM | Report abuse
Posted by: Linda/RetiredTeacher | August 5, 2010 8:58 AM | Report abuse
Posted by: lacy41 | August 5, 2010 9:09 AM | Report abuse
Posted by: interested8 | August 5, 2010 11:38 AM | Report abuse
Posted by: Wyrm1 | August 5, 2010 12:33 PM | Report abuse
Posted by: phillipmarlowe | August 5, 2010 12:54 PM | Report abuse
Posted by: TeacherSabrina | August 5, 2010 1:25 PM | Report abuse
Posted by: Incidentally | August 5, 2010 2:42 PM | Report abuse
Posted by: efavorite | August 5, 2010 3:05 PM | Report abuse
Posted by: bsallamack | August 5, 2010 4:01 PM | Report abuse
Posted by: bsallamack | August 5, 2010 4:43 PM | Report abuse
Posted by: DavidBCohen | August 5, 2010 7:37 PM | Report abuse
Posted by: bsallamack | August 5, 2010 8:14 PM | Report abuse
Posted by: edlharris | August 5, 2010 11:25 PM | Report abuse
Posted by: tutucker | August 5, 2010 11:41 PM | Report abuse
Posted by: tutucker | August 5, 2010 11:45 PM | Report abuse
Posted by: tutucker | August 5, 2010 11:51 PM | Report abuse
Posted by: tutucker | August 6, 2010 12:09 AM | Report abuse
Posted by: bsallamack | August 6, 2010 1:25 AM | Report abuse
Posted by: bsallamack | August 6, 2010 1:33 AM | Report abuse
Posted by: bsallamack | August 6, 2010 2:02 AM | Report abuse
Posted by: tutucker | August 6, 2010 10:51 AM | Report abuse
Posted by: tutucker | August 6, 2010 10:53 AM | Report abuse
Posted by: bsallamack | August 6, 2010 12:44 PM | Report abuse
Posted by: bsallamack | August 6, 2010 3:15 PM | Report abuse
Posted by: mport84 | August 7, 2010 9:25 AM | Report abuse
Posted by: bsallamack | August 8, 2010 3:35 PM | Report abuse
The comments to this entry are closed.