Network News

X My Profile
View More Activity


Posted at 5:00 AM ET, 03/ 8/2011

The ed report that all 'reformers' should read

By Valerie Strauss

Correction: An earlier version of this incorrectly reported that the study said that standardized test scores cannot be used to show progress in student learning. The study says that scores are of limited value in determining causes of improvements in student performance.

Here are two take-away messages for school reformers around the country that are embedded in an important new report about how the D.C. public school system is faring under the 2007 law that gave the mayor control.

* The standardized test scores that have been trumpeted to show improvement in the schools provide limited information about the causes of improvements or variability in student performance. This would be true, presumably, for any school system that use standardized tests as a measure of achievement.

*The accountability mavens who have pushed standardized-test-based accountability systems on schools, students and teachers forgot something important: to create and implement a system to hold themselves accountable.

The report, titled “A Plan for Evaluating the District of Columbia’s Public Schools: From Impressions to Evidence,” was issued Friday by the National Research Council, the research arm of the National Academies, which include the National Academy of Sciences, the National Academy of Engineering and the Institute of Medicine. It is the first in a series of mandatory reports under the Public Education Reform Amendment Act (PERAA), which allowed then-Mayor Adrian Fenty to take control of the city’s public schools and unilaterally appoint Michelle Rhee as chancellor.

My colleague Bill Turque wrote in a Pos story that reading and math scores on the District’s two main tests, the annual D.C. Comprehensive Assessment System (DC CAS) and the National Assessment of Educational Progress (NAEP), have shown what the report describes as "modest improvement" under Rhee and her predecessor, former superintendent Clifford Janey, although elementary-grade scores fell in 2010.

Rhee made the DC CAS the prime currency of her school reform effort to evaluate schools, students and teachers. But the report says that there are so many problems with the test scores that an overall rise in the numbers cannot be pointed to as proof of student learning.

The report says:
“1. The DC CAS is designed to measure students’ mastery of specific academic skills, but determining whether and how the changes in district policies or strategies have contributed to those skills requires additional empirical evidence: the scores themselves do not provide evidence about what accounts for them.

“2. The available scores are averaged across the entire student population, and do not provide information on the status or progress of specific groups: some may be making sharp gains while others are not.

“3. Because DC is a highly mobile district and the student population changes every year, score fluctuations may be the result of changes in the characteristics of the students taking the test, rather than improvements or declines in students’ knowledge and skills.

"Thus, in order to draw any conclusions about the effect of PERAA on student achievement as measured by DC CAS, further study of patterns for types of schools, individual schools, grade levels, neighborhoods, wards, and population subgroups is needed.”

This is an important finding because standardized tests have become the focal point of reform not only in the District but across the country, and the results are being used to evaluate not only kids but their schools and now teachers. Assessment experts have long said that these tests are not designed to do all of the things that are being done with them in the name of reform, but nobody in control of education policy has listened.

The report says: "Looking at test scores should be only a first step – not an end point – in considering questions about student achievement, or even more broadly, about student learning."

It also notes that reformers are happy to evaluate kids and teachers and schools, but not so much themselves. It says:

“PERAA was ... also spurred by impressions of the effectiveness of reforms in other urban districts facing at least somewhat similar economic, social, and historical challenges. Districts in Cincinnati, Minneapolis, New York City, Boston, and Chicago (among others) have focused on the alignment of content and performance standards with curricula, instruction, and other aspects of the school system. They have used data to guide their decisions, emphasizing such goals as improved professional development for teachers and principals; more frequent formative assessments; and the development of a culture of learning and collaboration among teachers. These approaches are widely used and are supported by some promising evidence, but the research literature is not yet settled enough to provide firm guidance on best practices for district reform or evaluation.

“Some districts have also focused on the governance of schools, and a few (e.g., Chicago, Boston, New York City and Cleveland) have given their mayors control over the public schools. Such reforms are designed to 'jolt' the system by changing dysfunctional institutional relationships and giving leaders new lines of authority and accountability. Evaluation of these governance reforms is critical to knowing what really works and what does not, but few cities have made this a priority, so there are neither clear exemplars nor substantial evidence to guide the District as it implements” the mayoral control law.

New York City, schools for example, have been under mayoral control since 2002, and that control has been extended to 2015. Still, the report indicates that the district is one of the school systems with mayoral control that failed to devise an adequate accountability system to evaluate the effectiveness of reforms undertaken under the administration of Mayor Michael Bloomberg.

Still, reformers proceed as if the reforms they have implemented have worked because test scores have gone up. This report lays bare the falsehoods behind this kind of thinking. We can only hope that somebody in control of education policy actually reads it and then gives it some serious thought.

Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed.Bookmark it!

By Valerie Strauss  | March 8, 2011; 5:00 AM ET
Categories:  D.C. Schools  | Tags:  dc cas, mayoral control, mayoral control of schools, michael bloomberg, michelle rhee, national academies, report on d.c. schools, standardized tests  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Why schools should try things not "research-based"
Next: Ravitch: Jon Stewart and other education heroes

Comments

Wow! I am so happy to see that someone else realises how and why there will be score fluctuation on standardised testing. I touched on this subject matter in my master's thesis.

Upon review of one school's recent national assessment testing results, it was discovered that the fluctuation in results may have other contributing factors. When examined closer, it was discovered that only 36% of the students who sat for the test were students who enrolled at the school in Transition and Year 1. The other 64% enrolled after Year 1, with the most students enrolling in the current year tested or the year prior.

In my study, there were also different degrees and speeds to which individuals adapt to new situations that are immeasurable.

Please could the reformers give some serious thought to what you are blaming and who you are holding accountable.

Posted by: GoldiLux | March 8, 2011 6:39 AM | Report abuse

Shortly after I discovered, by checking publicly available statistics, that DC's NAEP scores had been rising for over a decade, I started alerting reporters and making comments in WaPo articles.

The first comment I can find was a year and a half ago, on 10/30/09, in response to a letter supporting Rhee.
http://www.washingtonpost.com/wp-dyn/content/article/2009/10/29/AR2009102904378.html

Now we have a report from the national Academy of Sciences. Will that change anything, or is the myth of "significant gains" and "soaring scores" too firmly established?

Posted by: efavorite | March 8, 2011 6:55 AM | Report abuse

I'm happy to hear about your research, GoldieLux, and hope it gets some attention.

I'd like to see more people looking into what is behind the slow increase in scores over the last decade. Seems to me that if people really cared about the kids, they would do that.

Instead, the whole focus has been on the adults -- superintendents being praised for gains and teachers being held responsible for low scores.

Posted by: efavorite | March 8, 2011 7:55 AM | Report abuse

I can't wait to see what frankb1 will copy and paste in response to this one.

Posted by: MisterRog | March 8, 2011 8:41 AM | Report abuse

Why does it take a "study" to bring to the surface common sense?

Posted by: jdman2 | March 8, 2011 8:48 AM | Report abuse

As GoldiLux points, not all children can run at the same speed for the same distance. Playgrounds taught us since the beginning of recess, we just refuse to learn from the children.

What the reports say, essentially, is we don't know what we don't know. Our problem with standardized testing is not in the test, but in the use. If the test shows large discrepancies in schools/districts/states then they serve notice to educators to look at the results. If there is little separation, then maybe not so much but testing does serve a purpose IF used properly.

Tests might be mandated by politicians but tests themselves are developed by supposed educators. Who are they? What is their intent? How do we better align ourselves to get children learning instead of worrying about pass/fail, hire/fire, fund/defund.

Posted by: jbeeler | March 8, 2011 8:52 AM | Report abuse

Sorry Val. Didn't you hear the news? Gray is sticking with Kaya Henderson, and DC will continue the important and far-reaching reforms Michelle Rhee initiated.

Do some reporting on DCPS, you will discover dramatic "progress in student learning". Or don't. DCPS is moving forward with or without you.

The video below will make you laugh, cry and cheer. Much better than Jon Stewart (and of way more consequence).

From Henderson's Teach for America speech February 12, 2011:

"We are making history right here in the nation's capital...Some of you think it's over, it isn't over ..we're just getting started....this is the revolution that we all dreamed about...we went through a bloody battle to build a firm foundation so that we can provide these students with the education they deserve."

Full speech at: http://vimeo.com/19899601

Posted by: frankb1 | March 8, 2011 9:41 AM | Report abuse

Sure would be nice if we could get a whole list of small academic studies like Goldielux's for education policy makers to refer to and people like Valerie to report on.

As Jdman1 says and jbeeler implies, so much of it is common sense.

Unfortunately, the public and journalists, for the most part, have gone for the glitz and the miracle stories.

Posted by: efavorite | March 8, 2011 9:50 AM | Report abuse

Kaya Henderson on IMPACT:

"We don’t believe in basing a teacher’s pay only on test scores, but we do believe it is an important component when we measure a teacher’s effectiveness. … Nobody in the country has an evaluation system that is as sophisticated, that is as fair … and provides teachers multiple opportunities to show their stuff. When you change from a very simple system to a very complex system, there’s bound to be a lot of rumblings. … Our teachers are saying that they’ve never gotten this kind of specific feedback on their classroom instruction as they have with IMPACT.

The second thing is that IMPACT is allowing us to make decisions in completely different ways. … It has identified who our highest performing teachers are … it helps us spend our money differently. So, we spend millions of dollars on professional development. Previously it’s been random, with no idea as to what return on investment looks like. Now we can be proscriptive on what kind of professional development [the teacher needs]."

Full interview at:
http://georgetownvoice.com/2010/12/02/on-the-record-with-dcps-chancellor-kaya-henderson/

Posted by: frankb1 | March 8, 2011 9:50 AM | Report abuse

Kaya Henderson on IMPACT:

"We don’t believe in basing a teacher’s pay only on test scores, but we do believe it is an important component when we measure a teacher’s effectiveness. … Nobody in the country has an evaluation system that is as sophisticated, that is as fair … and provides teachers multiple opportunities to show their stuff. When you change from a very simple system to a very complex system, there’s bound to be a lot of rumblings. … Our teachers are saying that they’ve never gotten this kind of specific feedback on their classroom instruction as they have with IMPACT.

The second thing is that IMPACT is allowing us to make decisions in completely different ways. … It has identified who our highest performing teachers are … it helps us spend our money differently. So, we spend millions of dollars on professional development. Previously it’s been random, with no idea as to what return on investment looks like. Now we can be proscriptive on what kind of professional development [the teacher needs]."

Full interview at:
http://georgetownvoice.com/2010/12/02/on-the-record-with-dcps-chancellor-kaya-henderson/

Posted by: frankb1 | March 8, 2011 9:50 AM | Report abuse

Kaya Henderson on "changing the culture" of DCPS:

"We are engaged in a massive culture change … and culture change is like rerouting the Titanic. You can’t just turn a corner. The teacher is the lynchpin in that, but it’s [also] professional development, it’s curriculum … it’s change that has to happen in the classroom. … So keeping the culture change going and really pushing down to the point where the unit of change is the classroom is the biggest challenge."

Posted by: frankb1 | March 8, 2011 9:54 AM | Report abuse

Frankb1s' carefully sourced quotes on Henderson's support for IMPACT will come in handy once the scientific report on it is completed.

I'm guessing they will fall into the same category as all Rhee's statements about the soaring test scores she's experienced from her earliest days in Baltimore to her recent achievements in DCPS.

Keep it up, Frank - you're saving us a lot of work.

Posted by: efavorite | March 8, 2011 10:18 AM | Report abuse

Henderson on continuity & change:

"People keep asking me how I'm different from Michelle Rhee. I'm different than her because she's a petite Asian woman and I'm a large black girl," Henderson told The Washington Examiner."

Read more at the Washington Examiner: http://washingtonexaminer.com/local/dc/2010/12/battles-not-over-dcs-new-schools-chief#ixzz1G1OhWi6L

Posted by: frankb1 | March 8, 2011 10:19 AM | Report abuse

None of Frank's posts have anything to do with the article on any substantial level.


Posted by: DHume1 | March 8, 2011 10:28 AM | Report abuse

"We don’t believe in basing a teacher’s pay only on test scores, but we do believe it is an important component when we measure a teacher’s effectiveness."

FrankB,

These are the kinds of things supporters of the Pre-Nation at Risk (pre-1983) status quo are very threatened by and don't ever want to hear. Their reference to "deformers" for those of us looking to do more than rearrange the deck chairs on the Titanic, is their feeble, misguided attempt to stymie any and all reforms proposed to the way our school operated prior to 1983 and A Nation at Risk.

Yes, there are problems with some of the reforms (way too much testing and emphasis on the tests themselves) but something had to be done to improve our schools, SOMETHING.

I endorse what Henderson is doing in DC with IMPACT. It can be valuable for students, teachers and administrators to get a better handle on where students are and where they need help. Applause, applause.

Posted by: paulhoss | March 8, 2011 10:32 AM | Report abuse

efavorite wrote: "Henderson's support for IMPACT will come in handy once the scientific report on it is completed."

She's the architect of IMPACT, not just a "supporter".

Getting rid of hundreds of incompetent DCPS teachers has already had a positive effect. Hopefully Henderson fires hundreds more this spring.

From Kaya's bio:

"Henderson’s team also led the development of IMPACT, a new and innovative teacher assessment system designed to ensure that an effective teacher is leading every classroom in DCPS. Henderson’s work in developing human capital at DCPS has served as a model for other school districts throughout the country.

She has shared the successful strategies developed at DCPS with other districts and countries in national and international conferences."

http://www.dc.gov/DCPS/About+DCPS/Who+We+Are/Leadership+Team/Kaya+Henderson

Posted by: frankb1 | March 8, 2011 10:43 AM | Report abuse

paulhoss,

You lost me with the use of the "status quo" argument. It is easier for you to lump everyone together than it is for you to treat people in an "individualized" fashion. Yes, dude, you can't even follow your own pedagogic ideals.

Posted by: DHume1 | March 8, 2011 10:44 AM | Report abuse

frankb1, SIGH...still waiting for Rhee's comments on her students actual, demonstrated in class, growth. How she was transformed via the students reading chapter books, understanding number sense, improving classroom behavior so she no longer had to swallow bees or tape mouths shut.

As evidenced by your lack of quotes she has none. No direct "student's first" anecdotes regarding her own "transformation" Instead she relied on a principal's statement after her 2 (some say 3) years of teaching.

Wow...transforming, NOT

Posted by: researcher2 | March 8, 2011 10:53 AM | Report abuse

Someone suggested this video the other day:

http://www.talkingpointsmemo.com/archives/2011/03/now_were_talkin_1.php

To whoever that was, thank-you. It was great!

Posted by: jlp19 | March 8, 2011 11:48 AM | Report abuse

From Bill Turque:

Henderson said test scores remain the most objective available indicator of academic growth across the school system. "But I feel like we have to make people understand that test scores are not the only thing happening in our classrooms," she said.

"But I think we've gotten something wrong. Previously there was no measure of student achievement. We just sent kids to school and hoped for the best. And then the standards and accountability movement came along and said what doesn't get measured doesn't get done, so we have to test. And I think testing is incredibly important. But I also think that we have to help people understand that tests are a benchmark, not the goal. The goal is to educate children. And I think the swing of the pendulum from absolutely no accountability to what I might call data craziness is starting to hurt."

http://voices.washingtonpost.com/dcschools/2010/11/henderson_data_craziness_is_ta.html

Posted by: frankb1 | March 8, 2011 11:50 AM | Report abuse

The National Research Council’s evaluation and the Ginsburg study offer significant insights into the reform rhetoric and erroneous test results used to mislead policy makers and taxpayers for the benefit of the education profiteers.

Related to USA Today’s investigative reporting on cheating in DC and other urban school districts, how can fraudulent test scores be used to evaluate teachers, fire teachers and award merit pay? What if cheaters keep their jobs and those playing by the rules get pink slips?

It’s worth revisiting articles on cheating in DC -

http://voices.washingtonpost.com/class-struggle/2009/08/cheating_on_dc-cas_costs_chart.html

http://blogs.edweek.org/edweek/District_Dossier/2009/09/possible_cheating_in_dc_school.html

Will the deformers ignore the research and the recent investigative reporting?

Posted by: nfsbrrpkk | March 8, 2011 11:50 AM | Report abuse

DHume, buddy,

I'll try to make it easier for you in the future, you know, avoid some of the more complex analysis. That's code for keeping it simple (no, I won't say it).

Actually I wasn't lumping everyone together. The intent was to "differentiate" (get the reference?)the two status quo cohorts; the status quo group pre-1983 and especially pre- NCLB, versus the post-1983 status quo crowd. They are two different populations. Sure hope you can follow this.

Posted by: paulhoss | March 8, 2011 5:49 PM | Report abuse

paulhoss,

Yes, keep it easy for me, buddy, especially when you employ hyper-generalizations.

Whenever someone uses any form of the status quo argument, I am reminded of stupidity in all of its evolutionary forms. The problems that arise from its use range from false dichotomies, ad hoc unreasoned assumptions, and "common age" appeals. The entire argument usually balances on its use and displays a lack of rigorous acumen.

There are simple methods to get around this form of generalization and erroneous division, but I'm sure that you already know them all; and have good reasons for not using them.

Oh, BTW, I was also referring to your past posts regarding "individualized instruction" or "differentiated instruction." It is kinda ironic that someone who believes in this form of education employs "status quo" reasoning.

Posted by: DHume1 | March 8, 2011 6:33 PM | Report abuse

DHume,

Individualized instruction and any reference to the status quo are two very different beings.

Read the book. It's due out in May/June. You could learn something.

Posted by: paulhoss | March 8, 2011 9:09 PM | Report abuse


paulhoss,

"Individualized instruction and any reference to the status quo are two very different beings."

That was exactly my point, buddy: One generalizes about people and the other focuses solely on the individual. They are incongruous; they do not go together. However, they seem to find a nice, cozy home in that irrational brain of yours. Kinda says something about ideas and people don't it? Maybe it's not too late to put that in your book, too.

I like your point about individualized instruction, but how you get there and your rational for it is all caught up in a cyclone of contradictions.

Hopefully you have a good editor, buddy, to catch those little inconsistencies.

Posted by: DHume1 | March 8, 2011 10:27 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company