Network News

X My Profile
View More Activity


Posted at 12:35 PM ET, 11/ 2/2010

Rhee's testing legacy: An open question

By Valerie Strauss

This was written by Matthew Di Carlo, senior fellow at the non-profit Albert Shanker Institute, which is located in Washington, D.C. A version of this appeared on the institute’s blog.


By Matthew Di Carlo
Hardly anybody, regardless of their opinion about the newly departed D.C. Schools Chancellor Michelle Rhee thinks that test scores alone are an adequate indicator of student success.

But this is how the debate has unfolded, in no small part because of her own emphasis on them. Her aim was to raise scores and, with few exceptions (also here and here), even those who objected to her “abrasive” style and controversial policies seem to believe that she succeeded wildly in the testing area.

This conclusion is premature. A review of the record shows that Rhee’s test score “legacy” is an open question.

There are three main points to consider:

First, (by Rhee’s own admission) two simple policy changes enacted in 2007 were made, in part, to generate artificial test score gains during her first year (when roughly 75 percent of the DC-CAS increases occurred).

Second, the district’s DC-CAS test was introduced in 2006, and a year or two after any new test is introduced – as students, teachers, and administrators become more familiar with it – it’s common to see an artificial inflation in scores. The beginning of Rhee’s tenure coincided with this period.

Third, the students enrolled in D.C. public schools in 2010 were a significantly different group compared with the students of 2007, and this demographic shift may have driven some of the improvement in DC-CAS performance.

A deeper look at the best evidence we have – from the National Assessment of Education Progress (NAEP) – suggests that the increases in D.C.’s average NAEP scores between 2007 and 2009 (widely touted by Rhee and her supporters as confirmation of her effectiveness) could, in part, be a result of this demographic change. Math increases may be somewhat overstated, while reading scores may have been flat.

Policy changes creating artificial inflation
In a July 2009 Washington Post article, Bill Turque reported that, shortly after Rhee started, she made two bookkeeping changes that she knew would inflate performance during her first year, providing some invaluable political breathing room (she called some of these changes "low-hanging fruit").

First, she began enforcing a previously-unenforced policy that said high school students had to have enough credits to take the tests.

And second, she changed the way that students who don’t take the tests are recorded in the data (they were previously counted as failing, but Rhee changed the policy to exclude them from the data entirely).

Both changes resulted in groups of students being excluded from the data/test starting in 2008, but then results were compared with 2007, when both groups were included (the high schoolers without enough credits were almost certainly relatively low-scorers, on average, while the non-takers were previously counted as failing). These changes may have made sense for other reasons, but the fact that they inflated performance remains a serious problem when assessing DC-CAS results during Rhee’s tenure.

Without more data and details from DCPS, it is difficult to know how much these changes influenced the results.

We do know two things, however: First, because all year-to-year comparisons after 2008 would use consistent policies, the policy changes above would act to inflate results in 2008 only. And second, the vast majority of overall DC-CAS “gains” under Rhee occurred in 2008: 82 percent of the net increase in reading proficiency and 65 percent of the net increase in math proficiency occurred that year.

The new DC-CAS test and the “adjustment period”
The DC-CAS assessment system was introduced in 2006 by Clifford Janey, Rhee’s predecessor. (Full disclosure: Janey is a member of the Shanker Institute’s board of directors.) There is a good deal of evidence (here, here, and here, for example) that tests scores tend to falter or stagnate for the first year or two after a new testing system is introduced. This is usually followed by large score increases, after students, teachers, and administrators become more familiar with the format and content of the new exam.

D.C. test results followed this pattern exactly.

There is no way to know precisely how much of the score increases were due to the expected “adjustment effect” from the new test. But it’s safe to assume that some of the gains were. In the Post article discussed above, Rhee seemed to acknowledge this, saying that increases after 2009 would “absolutely” be real progress stemming from her reforms, implying that the previous years’ results might have been affected by the new policies. Talking about the first year during which the real benefits would show up (2010), Rhee said, “I’m very excited about next year.”

The 2010 DC-CAS results were largely flat.

DCPS students are different now compared with 2007.
Finally, the student population of DCPS changed rather dramatically between 2007 and 2010, with a large outflow of black students (the poorest and thus lowest-performing subgroup) and a corresponding rise in the proportions of students who are white, Hispanic, and Asian.

When you compare 2007 performance with that in later years, you are comparing unusually different groups of students, demographically speaking (like DC-CAS, most district data require "cohort-to-cohort" comparisons [e.g., elementary students one year compared with those the following year], but the overall demographic composition of students for an entire district tends to stay relatively stable over short periods of time). In the graph below, the somewhat unusual shift is clear.

graf one.png

[Note: Although this trend likely signals a net outflow of impoverished students from the district (black students have the highest poverty rates), there was no corresponding decline in the overall percentage of “economically disadvantaged” test takers. Indeed, there was an increase, from 62 percent in 2007 to 70 percent in 2010. It’s a safe assumption, though, that the recession was a factor in this shift. In other words, it seems likely that the increase in economically disadvantaged students was due to the reclassification of many of 2007’s “non-disadvantaged” students as “disadvantaged” between 2008 and 2010. A student’s race does not change over time, which means that the graph above reflects a churn of students, not classifications. Still - the outgoing students may have been relatively high performers (and/or not economically disadvantaged). These types of issues are always present when interpreting changes in cross-sectional data.]

We can’t yet tell how much this demographic shift inflated aggregate performance, but it’s entirely possible that it did, at least to some extent.

The evidence from the National Assessment of Educational Progress (NAEP) sheds a little light on the demographics issue. As an alternative to DC-CAS, DCPS frequently touts the results of NAEP as evidence of their reforms’ ”success” (though most of Rhee’s signature reforms, such as the Washington Teachers Union contract and IMPACT evaluation system, went into effect this year, and so had no effect on NAEP [or DC-CAS for that matter]).

A quick summary of the NAEP results for DCPS (public, non-charter schools) is presented in the table below.

graf three.png

Although eighth grade reading scores showed no statistically significant increase (i.e., they were flat), the gains in fourth grade reading and in math in both fourth and eighth grade appear to be impressive (though the NAEP increases had been occurring for several years under Rhee’s predecessors).

To get a rough idea of whether these improvements were real – or were, at least partially, a result of the change in cohort demographics (the shift was even stronger among NAEP test-takers) – we can check the changes in average scores for different subgroups. The simplified breakdowns by race are presented in the table below. Note that a separate breakdown for white eighth graders is not available in either subject, since there weren’t enough of them to get an accurate estimate (the same goes for Asian/Pacific Islander students in all grades/subjects).

graf four.bmp

As you can see, despite the claim in a piece that Rhee and Mayor Adrian Fenty wrote in the Wall Street Journal that “every student subgroup raised its performance,” the results were actually mixed. Although there were significant increases among black students in both fourth and eighth grade math, there were no discernible increases in reading for either grade. This is unsurprising in regard to eighth grade reading, where the overall results were also flat, but troubling in regard to fourth grade reading, where the widely touted overall gains are not shared by any subgroup.

There was, however, a significant increase in NAEP scores among low-income students between 2007 and 2009. This increase is noteworthy (black students tend to score lower because of poverty, parents’ education, and other non-racial factors), but it too may in part be a byproduct of the severe recession, with students who would have been above the low-income cutoff in 2007 coming in below it in 2009. NAEP (and published DC-CAS results) doesn’t follow individual students over time, so it is tough to untangle all of these factors.

As a result, there is no way to know the role that demography played in these results. Any time there are overall increases between cohorts, but none among any of the major racial subgroups, this is a red flag.

So, this is very tentative evidence that at least the fourth grade reading increases between 2007 and 2009 may have been, to some degree, artificial. On the other hand, the significant math increases among black students suggest that the overall math changes are real, though they may be somewhat overstated in the overall results, particularly for fourth grade (where scores for white and Hispanic students did not increase by a statistically significant margin).

***

Let’s quickly summarize. Three factors – surprisingly rapid demographic changes in DCPS students, a new state exam, and two changes to policies regarding who gets tested and how non-takers are accounted for in the data – are certain to have generated artificial increases in DC-CAS performance, even if we don’t know the extent.
We also know that this inflation was likely to have been especially strong in 2008 (when the majority of increases did occur), and especially weak in 2010 (when there were negligible increases and even decreases for some groups).

Finally, the same demographic changes that may have inflated the DC-CAS scores could have had the same effect on DC NAEP reading scores. The statistically significant increases in the NAEP math scores are also possibly overstated (though certainly still positive), due to this demographic change.

To all of this, add the fact that we don’t have any actual test score data from DC-CAS, and must rely on completely inadequate proficiency rates to assess performance (our only measure for students not in 4th or 8th grade, and for the years 2008 and 2010, when NAEP was not administered), and that, to my knowledge, there has not been a single even remotely sophisticated analysis of recent DCPS performance by an independent researcher with access to good data (an analysis, I might add, which could address many of the above issues).

So, it is fair to say that there are likely to have been some gains in achievement under Michelle Rhee’s tenure, but they were probably not dramatic, and certainly not unambiguous. The facts presented above strongly suggest that we should all sit back and wait for a more rigorous analysis of DCPS data before we issue any proclamations about Michelle Rhee’s testing “legacy.”

-0-

Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!

By Valerie Strauss  | November 2, 2010; 12:35 PM ET
Categories:  D.C. Schools, Guest Bloggers, Matthew Di Carlo, Standardized Tests  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: What other countries are really doing in education
Next: What Michelle Rhee did in D.C.: Point by point

Comments

Thank you. As a DCPS parent, I watched these trends play out firsthand. It wasn't inspiring.

I doubt your editors would allow it, but this really ought to be in the print edition of the post.

Posted by: Title1SoccerMom | November 2, 2010 2:38 PM | Report abuse

Good analysis. And just think, even if the scores held even, that would be the result of some real heavy lifting. If Dr. J. had remained in place, or if we had gotten someone who was yet another version of a puppet (with an ed. PhD) for the union, the scores would have been plunging, along w enrollment.

"How low can you go?" is the macabre catch-phrase for the DCPS. And since this is the District, no one is responsible for this decline, over several decades.

A new day is dawning, and the if the not-large cadre of rabid unionista teachers allow the level of chaos and disruption to decline and choose not to test the political will of Vince Gray, the Council, and parents, we just might get somewhere.

Posted by: axolotl | November 2, 2010 3:00 PM | Report abuse

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Yeah, the union must be the problem. They are the ones who are sending kids to school unprepared each day. They are the ones who don't spend the 20 minutes each day increasing reading fluency at home. They are the ones who could care less about passing a test or being educated. (Note the sarcasm.)

How long will people continue to make the union and teachers scapegoats for antiquated education policies? It's getting old.

Michelle Rhee was not the success story she was cracked up to be. The numbers (or manipulation of numbers) speak for themselves. As a result, she is gone.

Justice has finally been served. It's time to move on.

Posted by: syvetteavery | November 2, 2010 3:33 PM | Report abuse

Excellent! I too would like to see this in the print edition and otherwise widely distributed.

In addition, I'd like to see a piece exploring what factors might be responsible for the score increases under Rhee and past superintendents. Certainly it's shallow to declare that "Rhee did it" or "Janey did it" or all superintendents for over ten years have been doing it. That makes student achievement all about the adults, doesn't it? Isn't that exactly what Rhee and other so-called reformers are against?

Seems to me that if we really care about increasing student achievement we would start investigating how and why DCPS student achievement has been rising despite ongoing poverty and despite continuous changes in school leadership. A good analysis could help shape educational reforms that are based real-life experience and solid empirical research and could accelerate achievement gains here and in other high-poverty systems.

Posted by: efavorite | November 2, 2010 5:08 PM | Report abuse

Another great article telling the truth about Michelle Rhee. Now if only the mainstream media were as truthful.

Posted by: jlp19 | November 2, 2010 5:23 PM | Report abuse

axolotl: You really have a difficult time accepting the truth when it's right in front of you. Rhee's supporters hate data when it doesn't support her phony reforms.

As for unions: please change your tune. There was a great piece several days ago which cited data that shows there's NO correlation between unions and poor school performance.

Rhee and this wave of bogus public school reform is crashing fast and being exposed for the fraud they are.

Posted by: UrbanDweller | November 2, 2010 9:16 PM | Report abuse

This explains a little as to why so few of the publicly identified Highly Effective teachers are teaching subjects & students who take the DC-CAS.

Posted by: edlharris | November 2, 2010 11:50 PM | Report abuse

How convenient that you exclude charter schools from your analysis.

An alternative interpretation of your analysis is that those schools which employed unionized teachers showed the poorest test gains, and those schools which did not employ unionized teachers showed the highest test score gains. The natural and obvious conclusion is that teachers' unions negatively affect student learning, and we should rid ourselves of teachers' unions if we wish to enhance our children's learning.

Rhee and Fenty were correct - every student subgroup did improve on their test performance, when charters are included in the analysis. (It is well known in statistics that if you examine enough subgroups you will eventually find significant effects - this is essentially what you have done, by looking at the subgroups of DCPS students.)

Posted by: cypherp | November 3, 2010 10:01 AM | Report abuse

cypherp: I'm afraid your comment makes no sense. Michelle Rhee ran DCPS, so assessing her record requires that we look at DCPS. Charters are not relevant.

Posted by: logosmd | November 4, 2010 9:50 AM | Report abuse

Again! Unions donot hire teachers or protect ineffective teachers. Teacher evaluations have always been a part of the teaching process. Unions are like lawyers. They are there to protect the due process of its members like lawyers help you get justice. What about those school districts having the same problems with education where there is no union.Who do you blame? For the rich Education is new cash cow and "you" misguided commentors are clueless to what the agenda really is about. By the way what is education reform? Do we know it when we aee it? No! because there is no standard definition or model. Firing and overhiring is not reform. Defaming teachers and putting noneducators in the classroom is not reform either.Blaming unions is not reform. Using the cause for children is not the platfornm to use to further one's national platform where we are move on tv than doing the job the citizens pay you for. Rhee will crash and burn once she goes to another school district with her self incentive motivations.

Posted by: frankiesimmons1 | November 4, 2010 10:52 AM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company