Network News

X My Profile
View More Activity

D.C. test scores: chilling out about short-term changes

I told you so.

Way back on March 24, when the Post had headlines about Washington leading the nation in fourth-grade reading score increases, I recommended caution:

"We journalists have to write stories about these two-year score comparisons," I said, "because the federal government puts them out and readers have fun imposing their biases on the results. But they don't mean much. You need several years to determine if a change in education policy is working, and even then the results are usually inconclusive."

The same care should be applied to the news that average elementary school student scores went down this year on the D.C. Comprehensive Assessment System tests, but went up for secondary school students, compared with last year.

The debate over the scores is closely tied to the debate over D.C. Schools Chancellor Michelle A. Rhee.

I know it is confusing for me to say, as I did in March, that short-term test score changes, up or down, are not the best measure of her worth, when at the same time I say she is one of the best city school leaders I have ever seen. I am still one of her fans. Her energy and focus on raising student achievement is envied in cities such Detroit, where schools wallow in apathy. But all superintendents, including Rhee, are forced by city politics to give scores more attention than they are worth. We are seeing that again this week.

Overall in the District, the achievement trend is up since Rhee took charge. That is better than a downward trend. But we need to see how this goes for a few more years. After five years of any superintendent, you can make a fair appraisal. We aren't there yet.

I wish this were not an election year, giving test results far more political power than they deserve. There is nothing I can do about that. But I hope Mayor Adrian Fenty will reconsider statements like the one he made Tuesday: "Our secondary schools are doing fantastic, as are our elementary schools." He knows that is not true. He has been inside many of our secondary and elementary schools. He knows that the new results show that more than half of students are still not scoring in the proficient range.

Also, keep in mind that we are still comparing apples to oranges with this data. The students who took the tests this year are a significantly different group of kids, given the number of transients in any urban district, from the group who took the tests last year. Washington and the rest of the country are moving toward a new value-added way of reporting data that will give us the average improvement of individual children from one year to the next, rather than comparing Johnny's sixth grade this year to his big sister Janie's sixth grade last year.

Having said that, I confess I will be reporting on the test scores' gains or losses at some of the schools I have been watching closely, such as Coolidge and Dunbar high schools, as I did last year when I was watching Shaw Middle School at Garnet-Patterson. This will happen in August, after those scores are released and I'm back from vacation.

As I have said several times, I can think of better ways to judge schools than average test scores. I rate high schools each year based on participation in college-level tests, with no regard to the scores on those tests. I have written approvingly of proposals to rate schools by independent assessments of student work, or by regular inspections.

But for now, scores are the leading assessment tool. We education writers have to report on them, but we should do it very, very carefully.

Read Jay's blog every day, and follow all of The Post's Education coverage on Twitter, Facebook and our Education web page.

By Jay Mathews  | July 14, 2010; 12:45 PM ET
Categories:  Jay on the Web  | Tags:  comparing apples and oranges, last March's good news no better than this week's bad news, new D.C. tests scores require caution, politics intrude  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: What Bill Gates is really up to
Next: Students weren't taught research writing

Comments

Jay Mathews writes:
"But all superintendents, including Rhee, are forced by city politics to give scores more attention than they are worth."

That's a pretty passive construction to describe such a gung-ho, take-charge chancellor.

I think it was Rhee who led the charge to ignore all else, and separate good teachers from bad based on a formula that is comprised 50% by test scores....and not just this mythical "value-added" test scoring system, but whatever Johnnie this year and Janie last year deliver on the DC CAS each April.

I think it was Rhee who said, in effect, that test scores should be the be-almost-all in rating a teacher's effectiveness because anything else is just lazy teachers making excuses.

In fact, I'd be disappointed if we did not get a good breakdown by schools with Rhee-appointed principals, before and after the Rhee appointmnt was made, and compared to schools with experienced principals; students with all or mostly all TFA teachers vs. students with experienced teachers; and so forth.

Promises and good impressions don't count anymore in assessments of Rhee. Show me the data!

Posted by: Trulee | July 14, 2010 1:38 PM | Report abuse

Good points Trulee. Keep in mind that the IMPACT system, for all its flaws, is trying to use test scores to measure the individual growth of each child, not the apples and oranges comparisons of this year's district wide scores.

Posted by: Jay Mathews | July 14, 2010 1:44 PM | Report abuse

Ms. Rhee shouldn't be judged, at least at some level, on the performance of a sample of 30,000 kids? However, my co-workers who teach elementary school are being judged on a sample of 30 or so.

You can sit and say that one year variations in results shouldn't be the end of the world, and I agree. However, under IMPACT a lot of elementary school teachers are going to be found to be "below expectations" because of the one year performance scores of 30 (or less) kids.

Why shouldn't Ms. Rhee be responsible for the performance of her kids (all of DCPS)?

Posted by: Wyrm1 | July 14, 2010 1:47 PM | Report abuse

"We journalists have to write stories about these two-year score comparisons," I said, "because the federal government puts them out and readers have fun imposing their biases on the results. But they don't mean much. You need several years to determine if a change in education policy is working, and even then the results are usually inconclusive."
What? Standardized tests are inconlcusive? Did Jay Matthews really write this? Does Ms. Rhee know about this act of betrayal?
Did Jay also write:
"Also, keep in mind that we are still comparing apples to oranges with this data. The students who took the tests this year are a significantly different group of kids, given the number of transients in any urban district, from the group who took the tests last year."
What? The results can change due to the testing of different student groups? Wow, Jay. You better not let Ms. Rhee hear of such comments. She might try to slander you, the way she slandered some of the teachers she fired. Could it be that Jay has come over to the dark side (or the right side that believes that learning is too compicated to be captured by a standardized test). I would like to welcome Jay to our side, but Jay still can't stop drinking the Rhee kool-aid. Rhee is envied in Detroit? That's news to me. Jay it's sad that you realize the problems of the Rhee regime, but you can't bring yourself to separate from her. It's almost like you suffer from Battered Rhee Syndrome. You'll stay with Rhee, regardless of what she does to you. Jay, you remind me of the communists who supported Stalin, even though they knew he was murdering millions of his own people. How does that feel Jay?

Posted by: cincihenson | July 14, 2010 2:25 PM | Report abuse

“But all superintendents, including Rhee, are forced by city politics to give scores more attention than they are worth.”

Gee, I though it was Rhee herself who put so much emphasis on scores. I thought she prided herself on being data-driven. Did city politics force her to attach teacher evaluations to DC CAS scores?

“last year when I was watching Shaw Middle School at Garnet-Patterson”

You watched Shaw until their scores went down. Then I had to badger you for weeks (months?) to finally write about it and even then you tried to minimize the loss, until I badgered your editor after Rhee misspoke (again) in a Post on-line chat about the scores staying the about same.

“We education writers have to report on [scores], but we should do it very, very carefully.”

So why did you just repeat Rhee’s Baltimore miracle story of the 13th to 90th% score increase when she told it to you, without any evidence for it?

“"Our secondary schools are doing fantastic, as are our elementary schools." [Fenty] knows that is not true”

Finally, something we can agree on. What’s the hitch?

Trulee – Brandenburg analyzed scores by principals (for earlier years) back in February after Rhee bragged that "her" principals had rising scores. There were no existing tables that broke down gains by principal tenure, so Brandenburg developed them himself using available stats. Take a look:
http://gfbrandenburg.wordpress.com/2010/02/27/maybe-she-was-punishing-the-right-principals-not-so-clear/

Let’s see if Rhee brags again this year – without any way to prove it unless an industrious retired math teacher pulls together the data.

Posted by: efavorite | July 14, 2010 2:38 PM | Report abuse

I suppose there are different degrees of "fantastic" depending on the school system.

Posted by: fairfaxvaguy | July 14, 2010 2:49 PM | Report abuse

The Post reports that potential causes for the drop in elementary school test scores could be changes in personnel, shifts in student population, or modifications to the test. As a classroom teacher, whose evaluation and salary are so dependent on those scores, do I have any control over those variables? By the way, is there an independent organization out there anywhere that is monitoring how tests do change from year to year and how that affects what they purport to measure?

Posted by: izzy7 | July 14, 2010 3:42 PM | Report abuse

for efavorite--i always enjoy yr comments but i think there is some revisionist history there. I was always going to write about Shaw's test scores. I was just waiting for summer to end so there would be enough people around to read it. Now that the Post tells us (well, some of us, our education editor has been very good on prying this info out of the webmasters) how many page views we are getting, we have a much clearer idea of how much readership drops over the summer.

for cincihenson---your post did make me laugh, with you. very nice. some humor on the blog is always appreciated.

Posted by: Jay Mathews | July 14, 2010 3:49 PM | Report abuse

There is a great Mark Twain short story about how he came to edit an agricultural paper. I think the same lesson applies to some esteemed educational writers.

Posted by: mamoore1 | July 14, 2010 4:06 PM | Report abuse

These results are to be expected; when a school system aggressively hires individuals who have never taught then you add to the mix IMPACT and novice principals who are also learning on the job; a train wreck waiting to happen. I am all for hiring new teachers and principals but there are too many at one particular time, and the balance is way off. Let's take Shaw at Garnet Patterson; that school purged most of its veteran teaching staff during the first year of the late Brian Betts tenure. Well what happened, the test scores went down. Also add to the mix IMPACT, three learning styles in 20 minutes, with young children who need more time on task than a teacher putting up a show for the master educator or the principal.
This is a failing formula,
NOVICE PRINCIPAL + NOVICE TEACHER + IMPACT = LOWER TEST SCORES.

One more thing! If a failing high school under the definition of no child left behind gave every student an A.P course and most of the students got one's & two's and perhaps a few three's on the A.P. test would they be considered one of the best high schools in the nation? I will answer my question, "I don't think so". That happens in a D.C. high school that is a bit like Sousa Middle School, you should see the turn over there and read the "filthy teaching blog".

And one more thing! Scores since I have been in the district with different superintendents have never gone down. Well maybe I am wrong! Go to the DCPS web-site and look at the spin, they claim that test scores went up in the secondary level as a headline, when you read the report they wrote they mention in passing the decline of the elementary schools. It's a little like Stalinist Russia I wonder if some of the elementary school superintendents who drank the Kool-aid will be removed from the pictures at the web-site. There will be hell to pay at some of this elementary schools, expect some other elementary school principals to get sacked. With our lovely Chancellor expect the unexpected. By the way, the big first gains were done by Dr. Jeney, they are using these to offset the losses just like Enron; comparing his gains to her embarrassing losses. Remember he implemented the standards and Rhee flew on automatic pilot for that first year, all she was interested at was firing the persons in the central office. Romans12:19 seems fair to me.

Posted by: marylight | July 14, 2010 4:58 PM | Report abuse

This editorial is utter nonsense and political flackery at its best!

I actually agree with the substance--scores that are not value-added are useless and actually ineffective and misleading measures. The Chancellor and other educational reformers are smart to move to the value-added system. But, Jay--you are a proponent of scores-- that you are now disregarding because it's politcally convenient to do so...I have read multiple editorials by you citing the Chancellor's success with scores- whether it is AP scores or CAS scores--when it comes to your political motivation, you use it. And let me be clear--I am a fan of the measures being taken by the Chancellor--for the most part, that is-- I support the new teacher contract because people should be paid for the success of their teaching--after all, none of us would continue to pay a tennis instructor to give our children tennis lessons if he or she never learned how to play tennis. I support the new teacher evaluation system--because we need to define what effective teachers know and can be able to do. I support raising accountability for principals and administrators because the stakes are too high and America is falling behind the rest of the world in math, science and literacy. But, I do not support the unmistakable politcal flackery by newspapers across the country and other media outlets who use as support for their own political agendas the misuse of news and information reporting.

This is dangerous- very dangerous. Every night on news outlets I am disheartened by the biased news reporting of MSNBC- liberal and FOX- conservative. So many Americans are being used by media outlets for political posturing--it is as sickening for conservative tea party activists to place Obama in a picture next to Hitler as it is for liberals to suggest that all tea party members will use guns to reform goverment. And while this may seem like a tangent- it is not. The same thing is happening with the editorials of the Post with regard to Rhee. Yes--editorials are opinions, but most people don't distinguish and this editorial is a reflection of that. For a journalist who judges the successes of schools by how many students take AP (an absolute idiotic standard given that there are so many more measures that dictate student success)this editorial is disingenuous at best and unethical at worst. Though, I agree with the substance of your points--you have done a 180 on your positions about student achievement to serve your political support of the Chancellor. I would rather you have written, "Maybe, I was wrong to tout every successful achievement over the last 3 years in testing that clearly had as little to do with this Chancellor's reforms as this year's scores do with her failures" -but you, didn't. Instead, you attempted to spin your own viewpoints to protect the Chancellor from criticism. No need for this.

We're headed in the right direction under the Chancellor- just say that.

Posted by: teacher6402 | July 14, 2010 5:16 PM | Report abuse

Jay isn't the only one at the post suffering from "Battered Rhee Syndrome."

Posted by: aby1 | July 14, 2010 5:24 PM | Report abuse

She should be judged by the same standard as she's judging teachers. If scores fall, she should be fired. What's "fair" for teachers should be fair for chancellors!

Posted by: UrbanDweller | July 14, 2010 5:35 PM | Report abuse

Jay, it is still so curious to me (despite your explanation a couple of weeks ago) that you hold Rafe Esquith up on one hand and Michelle Rhee on the other as exemplars. MR has said flat out that "individual" or heroic teachers (like Rafe) don't matter in the big scheme of things as their overall effect is minimal. The system is what matters. Rafe, OTH, proves that "individual" teachers are who gets real results despite all the forces of mediocrity in public education. He would probably have many issues with the Rhee-gime if he taught here.

Could it be that more teachers could get "Rafe-like" results if top-down, test-driven administrators were few and far between? Or, are Rafe-like qualities rendered irrelevant by astute and omniscient administrators and politicians?

Still having trouble resolving your extremes, though the complexity of the education challenge, especially at the macro level, certainly requires two, possibly more, minds around it.

Posted by: thetensionmakesitwork | July 14, 2010 5:40 PM | Report abuse

Jay Matthews,
Isn't Dunbar HS being run by an outside entity from NYC? If so, how are they progressing? I'm especially interested in learning how attempting to turn around Dunbar compares to turning around similar schools in NYC. Word on the street is that not all urban schools with disadvantaged students are alike and that "DC's unique culture" creates more problems than in other cities. How true is this belief? I'd look forward to you tackling some of these questions.
A.L.

Posted by: lipschitzantwon | July 14, 2010 5:54 PM | Report abuse

Jay says, "for efavorite--i always enjoy yr comments but i think there is some revisionist history there"

I call projection - You are the revisionist, not I. Even teacher6402, who generally agrees with your point of view, notices it.

As I pointed out on another blog of yours, you did something similar in an article about the achievement gap - saying how immaterial it is.
http://voices.washingtonpost.com/class-struggle/2009/12/why_i_have_no_use_for_the_achi.html

I tend to agree, but Rhee doesn't and your column came out just after it became known that the gap had widened under Rhee.

Posted by: efavorite | July 14, 2010 6:00 PM | Report abuse

I hope you will do so, but please make a point to discuss the scores of Sousa and CHEC.

As for:
"The students who took the tests this year are a significantly different group of kids, given the number of transients in any urban district, from the group who took the tests last year"

Doh!

Posted by: edlharris | July 14, 2010 6:38 PM | Report abuse

I agree that there are a lot of transient students, and concede that it is possible that this caused a drop in scores.

However, on the NAEP we saw an increase in affluent primarily white students, which led to a fair proportion of the gains on that exam. So it seems that the "transient" students are actually helping the scores, not hurting them.

However, this should be easy to check. I'm sure DCPS will be happy to release the "value-added" of students in the system. They will certainly have it, since they are evaluating teachers on it. I'd like to see that data, since then we would be comparing apples to apples. I suspect we will see it if it is beneficial, and not if it is not.

Posted by: Wyrm1 | July 14, 2010 7:31 PM | Report abuse

If you are a current DCPS looking for a position but not having much luck, you can check for jobs through this advert through ASCD:
https://jobs.smartbrief.com/action/listing?listingid=CD727267-3C9B-46F4-B504-16B3A44FE3D5&briefid=3E572E12-3FBC-11D5-AD13-000244141872&industryid=68DAF4D1-6B91-4720-9427-1CD7744D8012

Posted by: edlharris | July 14, 2010 8:08 PM | Report abuse

Thank God for GF Brandenburg's blog site. I find it strange that in order to get factual statements and thoughts concerning educational issues, a reader has to go to a retired math teacher instead of a "educational journalist". Perhaps this is why most Americans don't trust the media or journalists.

Posted by: cincihenson | July 14, 2010 10:12 PM | Report abuse

Thank God for GF Brandenburg's blog site. I find it strange that in order to get factual statements and thoughts concerning educational issues, a reader has to go to a retired math teacher instead of a "educational journalist". Perhaps this is why most Americans don't trust the media or journalists.

Posted by: cincihenson | July 14, 2010 10:14 PM | Report abuse

Jay has consistently said that 1) Value-add data is better than averaging data and 2) it takes several years worth of data to make a realistic appraisal of progress.

It's interesting to me that so many commenters accuse him of being a shill for Rhee in his comments here when the guy made the same point back in March when the data was more favorable. Plus, it's not exactly like the guy is hiding his biases. He's quite straightforward in terms of his opinions on education. I'm confident readers are smart enough to decide for themselves what his opinions are worth.

Posted by: Jessedavidam | July 14, 2010 10:29 PM | Report abuse

Detroit schools suffer from poverty, not apathy.

Posted by: celestun100 | July 14, 2010 11:23 PM | Report abuse

As JM is neither shy nor retiring, he might petition his editors to have him join some schools inspection teams, or even get some basic training from a quality education auditing company, so he learns what those teams are looking for. Yes, he's written here about a serious ed policy wonk's advocacy of something as extensive as the British Ofsted
[www.ofsted.gov.uk] model of inspection. But isn't it likely that some Catholic dioceses with large numbers of schools have used externally trained and professional inspection teams to impartially audit their schools; and that closer to home the Public Charter School Board has employed similar teams? On the latter, it would seem so from the write-ups on many of the schools, as revealed in the most recent Annual Report, where remarks would seem to be drawn from longer reports.

I find no mention of those studies or audits reports and the methods they employ at the PCB web page. Maybe inquiry by the blogger would inform him and then us of how the PCB goes about uniformly conducting its education audits of schools.* The charter-friendly and charter-supportive Fordham Institute is keen on and has performance guidelines for schools it has co-sponsored.

*The motivator for the DC CAS under discussion is not the evaluation of students, but the certification of schools as meeting standards of Adequate Yearly Progress. As for meeting presumptive standards of Adequate Teaching, there's nothing to stop the blogger from trying to persuade the WaPo data-based research team from collecting from DCPS the pooled IMPACT evaluations, anonymous for teachers, but identified by DCPS school.

Posted by: incredulous | July 15, 2010 5:01 AM | Report abuse

Jay, I actually agree with you about overall performance over years and test scores. My issue that that Ms. Rhee has herself been one of the strongests advocates for establishing a direct correlation between acheivement scores and assesssments of effective teaching.

Now Rhee has to figure out how to sustain her positions on teacher pay and evaluations in the light of oddly lowered elementary scores in both reading and math. That said, it's great to see so many researchers and education writers coming out to say, "yes, in any given cohort of students, year to year peaks and valleys are to be expected. Continuous trends up ward are unrealistsic." That rational approach has been in short supply, particularly from Michelle Rhee.

I have no sympathy at all for her.

Posted by: Nikki1231 | July 15, 2010 8:13 AM | Report abuse

for teacher6402---I hope you are not assuming that I write the education editorials on our editorial page. I do not. I share some opinions, but not all, with that person, who is a former boss of mine. I challenge you to post all these comments you say I made extolling Rhee for her great test scores. I haven't done that. And I definitely haven't changed my views on achievement. I have made much of test score trends that show long term individual student growth, like the KIPP national results. That is not what I am talking about here.

Incredulous---very good suggestions. I vaguely recall asking the DCPCSB about those inspection reports, as something that would eventually be available, when I was doing a column about bad charters in DC. I will pursue this.

Jessedavidam---thanks for noticing. This blog won't work unless I am very clear about my biases (I actually believe I only have a few, all backed by research) and unless readers scream when I say something I cannot back up.

cincihenson---I share your admiration for Guy's blog. That is why I named it best in DC.

edlharris--good suggestion. I suspect Bill Turque or Stephanie McCrummen will reveal the Sousa results before I get back from vacation, and my week moving my office from the Alexandria buro to my home, but we will reveal them to readers in some way, and if we don't let me know. I hadn't thought about Columbia Heights Ed Center as a place to watch for DC CAS, but it is also a good idea.

marylight---If you are comfortable rating high schools only by test scores, which is actually rating them by the size of the parents' bank accounts, that's fine with me. You are in the majority on that. I prefer to rate what the school is doing, not how much money the parents are making. If you have a school full of low income students whose teachers are taking the rare iniative of exposing them to AP and raising the level of instruction, that is a useful measure of the school.

lipschitzantwon---That is exactly right. A group of NYC educators are running those two high schools, and I will report on their results next month. Click on the link in this column to Coolidge and Dunbar and you will get my latest column on their efforts.

Posted by: Jay Mathews | July 15, 2010 11:57 AM | Report abuse

Dear, Mr. Mathews

I agree with you that students should be exposed to AP courses in low income schools. I also agree with you in when you said you said, "I prefer to rate what the school is doing, not how much money the parents are making". There is a correlation between tests scores and parental income. I hope the Chancellor understands this correlation because she has said that "poverty is not an excuse". I strongly disagree with ranking schools solely on how many students take AP tests regardless of outcome. There is a particular high school in this city that understands this scheme and uses this formula to rank as one of America's best schools every year. In reality this school should be called the Valley of the Fallen for its a monument to its leader and all the teachers who leave there every year bitter.

Respectfully,
Marylight

I would give you my real name and my official title but in this climate it is too dangerous, it is better to pretend that you are drinking the Kool-Aid.

Posted by: marylight | July 15, 2010 4:40 PM | Report abuse

The numbers are down, what, 4%? Clearly Rhee has failed! Time to reinstate everyone who was ever fired from either schools, or the central office, and return to the good old days of the 80s and 90s. Great plan.

Anyway, scores will go up long-term as gentrification continues...

Posted by: ibc0 | July 15, 2010 5:24 PM | Report abuse

for marylight--thanks for the very thoughtful message. My email address is mathewsj@washpost.com. We both know which DC school you are talking about. You are telling me things I did not see when i visited the school, and have heard from none of its teachers. I would love an email that tells me what you know and how you know it, on any basis with which you are comfortable.

Posted by: Jay Mathews | July 15, 2010 5:28 PM | Report abuse

"If you are comfortable rating high schools only by test scores, which is actually rating them by the size of the parents' bank accounts, that's fine with me. "

It's true that test scores reflect the parent's bank account. But did you come to this realization on your own? Or are you defending Michelle Rhee in the light of the test score decrease?

Your understanding of the tie between parent's income and test scores came right at the time it was announced that elementary test scores had fallen. So it's hard to know whether or not the two are connected.

Posted by: jlp19 | July 16, 2010 11:29 AM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company