Network News

X My Profile
View More Activity

Better one or better two?

PH2010082203194.jpg

One of the policy mistakes we make a lot is to judge ideas in isolation when they need to be looked at comparatively. "Is this health-care bill perfect" is not as good a question as "is this health-care bill better than the current system and its plausible competitors?" Similarly, "does this system for evaluating teacher quality have problems" is a worse question than "does this system for evaluating teacher quality do a better or worse job than the status quo and its plausible competitors?" David Leonhardt got at this nicely in his column this week:

Value-added data is not gospel. Among the limitations, scores can bounce around from year to year for any one teacher, notes Ross Wiener of the Aspen Institute, who is generally a fan of the value-added approach. So a single year of scores — which some states may use for evaluation — can be misleading. In addition, students are not randomly assigned to teachers; indeed, principals may deliberately assign slow learners to certain teachers, unfairly lowering their scores. As for the tests themselves, most do not even try to measure the social skills that are crucial to early learning. The value-added data probably can identify the best and worst teachers, researchers say, but it may not be very reliable at distinguishing among teachers in the middle of the pack. [...]

[But] we will have to acknowledge that no system is perfect. If principals and teachers are allowed to grade themselves, as they long have been, our schools are guaranteed to betray many students. If schools instead try to measure the work of teachers, some will inevitably be misjudged. “On whose behalf do you want to make the mistake — the kids or the teachers?” asks Kati Haycock, president of the Education Trust. “We’ve always erred on behalf of the adults before.”

You may want to keep that in mind if you ever get a chance to look at a list of teachers and their value-added scores. Some teachers, no doubt, are being done a disservice. Then again, so were a whole lot of students.

Policy questions are like going to the ophthalmologist: Better one or better two?

I'd also echo another point Leonhardt makes: I'm sympathetic to the idea that the L.A. Times's decision to partner with an education economist, do a quick-and-dirty analysis of test scores, and then release the data next to the teacher's names is a bit of a radical step. Parents are going to see that data, and it really is likely that they're going to get the wrong impressions about a lot of the teachers (though they'll also get the right impressions about many teachers). But as Leonhardt says, "one way to think about the Los Angeles case is as an understandable overreaction to an unacceptable status quo."

The teacher's unions really do need to figure out a way of assessing teacher performance that's acceptable to them, because in its absence, they're going to get steamrolled by solutions that aren't acceptable to them.

Photo credit: Ricky Carioti/the Washington Post Photo.

By Ezra Klein  |  September 3, 2010; 5:40 PM ET
Categories:  Education  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: The jobs numbers and uncertainty
Next: Reconciliation

Comments

"Some teachers, no doubt, are being done a disservice. Then again, so were a whole lot of students."

Are students not done a disservice if compentent teachers are dismissed based on aa badly flawed measurement, to be replaced by the latest 'flavor of the month' Teach for America or other idealistic but untrained newbie?

Most of the arguments about getting rid of ineffective teachers have the tacit assumption that somehow there is a vast pool of willing and effective teachers 'out there' somewhere, being held back from rushing into the classroom and rescuing the suffereing, ill taught kids who would all be Rhodes Scholar candidates if only they had the right teachers. No one wants to face the fact that if we want better students, the only proven way to get them is to figure out how to help parents be better parents. And the chances of doing this in our increasingly polarized 'ownership' (i.e., 'you're on your own) society are pretty much nil.

Posted by: guesswhosue | September 3, 2010 6:31 PM | Report abuse

First: guesswhosue nails it.

Second: I think it's kind of insulting creating the false dichotomy you do: teacher/administrator-led evaluations vs. value-add mechanisms. Even if you insist upon eliminating teacher/admin.-led evaluations, leaping to the conclusion that value-add metrics that rely on some deeply inconclusive test data is just lazy.

I think most critics of value-add evals aren't saying "Stick with admin/teacher evals," I think they're saying, "get it right." If you replace something bad with something only marginally less-bad, you haven't replaced it with something good.

Posted by: falsedichotomy | September 3, 2010 7:30 PM | Report abuse

Incidentally, I teach in an urban high school with 70% of students on free or reduced lunch, and a teaching staff that is 75% unionized (myself included). After having earned AYP for three consecutive years, we are beginning to receive both local and national attention. In short, we are Arne Duncan's absolute worst nightmare: a public school with primarily union employees getting a graduation rate of 90% and a college acceptance rate of the same percentage (with very low college attrition). Will our school get a visit from Arne? Don't count on it: we're not charter. We don't fit the beltway groupthink. In effect, we don't exist. We'll continue to be great, nevertheless. Because our staff is second to none and our kids know we love them and have their success at our heart's core.

Posted by: falsedichotomy | September 3, 2010 7:56 PM | Report abuse

Ezra and Matthew Yglesias seem to have somehow been convinced to tow the beltway party line. Though their performance, especially Ezra's is not very convincing.

The only argument I can see in favor of the (on the face of it) stupid and misguided "race to the top" is that it is a prelude to a secret plan of sometime in future changing the test to make it more reasonable and curriculum based, with the ultimate goal in the distant future of having a the competition for federal funds serve as an incentive for schools conform to a nationwide uniform curriculum. A cynical plan that will hurt a lot of teachers and a generation or so of teachers and students in the short run if it ever works in the long run. And when I say hurt I mean ruin the lives of. It is a strategy suitable for war, perhaps but violates the "first do no harm" ethic that physicians and other social helpers ought to adhere to.

Posted by: harold3 | September 3, 2010 9:11 PM | Report abuse

Like many of your generation, Ezra, you are easily fooled by proffered technocratic solutions. Isn't this what's doing Obama in?

Don't ask is one better than two, ask who is proposing one and why.

Even assuming that you could create a truly objective measure, that is, one that did not have, like all technologies and systems, a built in bias or politics (and think who will do the designing of the measures) your one dimensional take on assessment ignores the ways in which it changes the very essense of education. It reduces it to precisely the sort of thing that can be measured. Teaching to the test is only the most obvious cost here and not the most profound. Far greater is the transmutation of education from a humanistic experience that shapes whole beings into a set of skills more or less demanded today, but soon to be obsolete.

The idea that teachers, individually, transmit (what, excatly--knowledge, informaiton, skills; or perhaps just performance on your test, for a perfect tautology) and students, individually, receive, and that improving the first improves the second ignores the entire social context of education. There are ways of improving education, but they would require paying money and promoting teaching as a desirable profession.So instead we get a quick and seemingly costless answer--just measure better.

The measurement and assessment crowd is tinkering at the margins when the problems of education are far deeper, ranging from underfunding to the debasement of teaching as a profession, to the lack of parental engagment with the intellectual development of their own children. Blaming bad teaching is the easy out.

Posted by: groundedlogic | September 3, 2010 11:59 PM | Report abuse

I think it's good for public employees that they have a union that can protect their rights against arbitrary dismissal or if they have problems in their life affecting their job performance. However, they should be removed from teaching during the time period that they're getting themselves straightened out after serious family issues, emotional strain from a messy divorce, substance abuse or drinking, etc., so that it doesn't have to affect the children they teach. Yes, other people in private companies don't have such rights, but just because private companies don't treat their long-term employees decently doesn't mean that public employees should be treated like dirt and thrown away like a used Kleenex also.

Posted by: stillaliberal | September 4, 2010 11:23 AM | Report abuse

I think it's good for public employees that they have a union that can protect their rights against arbitrary dismissal or if they have problems in their life affecting their job performance. However, they should be removed from teaching during the time period that they're getting themselves straightened out after serious family issues, emotional strain from a messy divorce, substance abuse or drinking, etc., so that it doesn't have to affect the children they teach. Yes, other people in private companies don't have such rights, but just because private companies don't treat their long-term employees decently doesn't mean that public employees should be treated like dirt and thrown away like a used Kleenex also.

Posted by: stillaliberal | September 4, 2010 11:23 AM | Report abuse

Good God. Plainly, the apocalypse is near.

Leonhardt is one of the brightest guys in the mainstream press. But "among the limitations" of the value-added method (he only cites two), he includes this supposed limitation:

"Students are not randomly assigned to teachers; indeed, principals may deliberately assign slow learners to certain teachers, unfairly lowering their scores."

The "value-added" method was specifically designed to allow for that type of situation. Indeed, that is precisely why the system was invented.

To all appearances, Leonhardt has no idea how "value-added" actually works. (I say that after reading his entire piece.) Apparently, neither does Ezra--another of the bright guys.

Does anyone give a flying fig about public schools? The question keeps answering itself.

Posted by: bobsomerby | September 4, 2010 11:49 AM | Report abuse

"The teacher's unions really do need to figure out a way of assessing teacher performance that's acceptable to them"
Years experience is the only one they care about. (It's a good proxy for total union dues contributed) Haha. My dad was in the teacher's union for a while. I think he dropped out when they decided to strike in the middle of the school year, hurting a lot of his kids chances to succeed. Of course, he had three other jobs at the same time, so I am guessing pay was a big issue back then. :-)

Value-add is infinitely better that just looking at where kids end up, without considering the varied places where they start. You can enhance it by comparing it to the progress the same kid made in other years. The utter stupidity of NCLB, the ugly half-brain-child of Bush II and Ted Kennedy, was obvious to me before it was even signed. Evaluating schools by the raw performance of their students is brain-dead, even for politicians.

Of course, even have a standardized raw number is a good measure for parents to look at as they attempt to find a school with students that match their child's ability. The inanity of a system that puts kids into classes for their elementary years without regard to their current level makes many people question what purpose the schools are being twisted into. Differentiated classes can start before kindergarten.

I fail to understand the "no standardized tests" people. It's not just multiple choice anymore people, the AP tests are some of the best out there.

Posted by: staticvars | September 4, 2010 2:30 PM | Report abuse

Publishing teachers' value-added rankings in the paper strikes me as roughly parallel to the practice of parading suspected prostitutes through the streets in China. Let's shame people into improving! And if some of them aren't actually prostitutes, well then it's still a small price to pay for the greater social good.

But even if the rankings were one hundred percent reliable it would still be a terrible idea: once the rankings come out, who's going to get which teachers? The children of dedicated, well-informed parents with the leisure time to hound the administration will get the good teachers, and the children of poor, uninformed parents will get stuck with the crappy teachers, exacerbating inequality.

If we want a corps of uniformly excellent, dedicated teachers in this country, we need to stop acting like you can punish people into excellence. Withdrawing funding for schools that are failing? We should be doubling their funding; we should be sending failing teachers to learn from other, more successful teachers; we should be offering good teachers at good schools a 20% raise to transfer to a struggling school. If a teacher is bad then *teach them how to be a good teacher,* don't just stand there beating and haranguing them.

Posted by: heresiarch | September 4, 2010 2:30 PM | Report abuse

Teachers' performance is assessed. In my state there are teaching standards that have to be met by the teacher and evaluations are based on those. People seem to think that once a teacher is hired, that's it. It's easy street until retirement, that there is no oversight or accountability. In fact, the opposite is true. Though I've taught over a dozen years, I go through a rigorous evaluation process every two years. To say that unions must agree to something ignores the fact that something is already there.

The fact is, teaching can't be evaluated by a number, convenient though it is. Test scores are an inadequate measure of how well students are learning and teachers are teaching. First of all, they don't necessarily reflect what's been taught (standards vary from state to state). Secondly, the students face no consequences for doing poorly on the tests. They have no bearing on grades or promotion and students know this. Is it fair to base teacher pay and career on whether or not a student had breakfast that morning? or was bored and tried to make a pretty pattern on the bubble sheets?

I am very uncomfortable with how the LA Times published teacher names along with VAT scores. As an employee, I expect a bit of privacy when it comes to matters of evaluation and employment. In fact, districts that do use VAT keep teacher scores confidential, as well they should.

Posted by: lkayed | September 4, 2010 4:43 PM | Report abuse

>"Students are not randomly assigned to teachers; indeed, principals may deliberately assign slow learners to certain teachers, unfairly lowering their scores."

>The "value-added" method was specifically designed to allow for that type of situation. Indeed, that is precisely why the system was invented. - written by bobsomerby

Although I support using value-added measures, Leonhardt does actually have a point. However, almost all commentators do not understand the subtlety.

Because of lack of randomization, there can be "confounding variables" between teachers' classes. A glimpse of these confounding variables can be seen by looking at the effect of 5th grade teachers on 4th grade test scores. Intuition tells us that there should be no effect - events in the future cannot affect the past. However, some value added methods show this effect of future teachers affecting past test scores.

Let me give another example. A company develops a red wine extract and claims it prolongs life. They test the extract on 2 groups of people, a control group (who doesn't receive the extract) and a treatment group (who does receive the extract). The treatment group lives significantly longer. After the experiment, someone notices that the treatment group was significantly younger than the control group. Consequently, the results of the experiment are called into question. Next time they do the experiment on randomly chosen groups of approximately equal age, and find no difference in outcomes.

The LATimes method paper I believe calls these confounding variables endogenous unobserved variables, or something like that.

All that being said, VA methods seem to be at least as good as any other method.

I hope, through the prominence VA methods have acquired lately, that we can agree nationally on a common definition of "effectiveness" in teaching, and start delineating what makes an effective teacher.

Posted by: cypherp | September 4, 2010 6:40 PM | Report abuse

If VA methods are just as good as others, then other methods are just as good as VA. Teachers are evaluated - constantly. Look, if you want to weed out ineffective teachers, then what we need to do is to take the evaluation process out of the hands of principals. Principals are very, very busy people. A strong principal will work to transfer out the ineffective teachers and/or ineffective teachers choose transfers to stay one step ahead of the evaluation process. If evaluations and follow-up measures were handled at the district rather than site level, or perhaps an outside accreditation organization, this kind of cat and mouse couldn't and therefore, wouldn't happen. When contemplating changes to the teaching profession, let's not make it less attractive than it is. There is a teacher shortage brewing that is currently masked by the recession. As commenter #1 pointed out, it's not like there are masses of young college grads champing at the bit for a teaching job.

Posted by: lkayed | September 4, 2010 7:58 PM | Report abuse

Let's get down to brass tacks here:

Cut teaching staff by 25% - we can easily say the bottom 25% aren't worth having around. If you think they are, you are the problem. I repeat, anyone who disagrees that 25% of our teachers should not teaching - you have no grasp of reality.

Increase class sizes by 1/3 - ALL parents would prefer their kid be in a class 33% bigger if they are GUARANTEED their kid doesn't get taught by a teacher in the bottom 25%.

Everyone knows in their heart that if your kid gets the bottom 25% of the current crop - you might as well give them cancer.

Pay the remaining 75% of the teachers, another 10% bonus in salary - but funnel it all to the top third at the end of the year.
Crush any union that argues.

Track each kids scores, use th LA method - there will be some mistakes made, but it's worth firing the bottom 25% immediately.

Posted by: in2liberty | September 4, 2010 11:05 PM | Report abuse

So far, no one from Klein on down has offered the only solution that will absolutely work: Shut down the government schools. You all can blather until doomsday about every other possible permutation and you're still going to be looking at failing schools costing ever and ever more. The failure is in attempting to jam all your stupid eggs into the corrupt, incompetent, political government basket, and nothing done within that framework is ever going to amount to anything anyone will ever be proud of.

Posted by: msoja | September 4, 2010 11:10 PM | Report abuse

Wow, in2liberty, that's brilliant! Because once you get rid of the bottom 25% of current teachers there won't be a new bottom 25%percent of teachers for parents to worry about their kids getting! And larger class sizes--astoundingly insightful! After all, who has ever suggested that larger class sizes hurt learning?

It's totally not at all like you're suggesting a set of changes guaranteed to destroy the public education system of the United States.

Posted by: heresiarch | September 5, 2010 12:16 AM | Report abuse

Here's where the money goes, at least in the district I work in.
Less than 1/3 to general-ed teaching salaries and all teaching supplies
Over 1/3 to the many, many individuals whose jobs consist of ensuring compliance with the Individuals with Disabilities Education Act: counselors, special education teachers and administrators, speech, OT, child study team (fills out IEP paperwork), nurse, and tuition to private schools for special needs children.
About 1/3 to health care benefits, principals, and building maintenance

Discussions of education are often framed comparatively: our nation lags behind X and Y. Blame is often placed either on teachers or on more general societal variables. But if you just follow the money, you can see that we are just valuing different things than other countries. It's not that special education funding doesn't accomplish some things-- I have a cousin with autistic-spectrum symptoms who went from American to Dutch schools, and I'm pretty sure he would have been better off in the States. But if I'm not sure our current system meets the greatest good for the greatest number test.
IN the case of Los Angeles, I'm all for teacher accountability, even by Value Added measures, but there's also something screwy about a city where the AVERAGE class size in high schools is 43 (meaning there are as many classes with 50 kids as with merely 35) publicly shaming teachers. Lastly, it should be remembered that, assuming a normal, bell-curved distribution of teaching ability, the teacher in the 17th percentile and the teacher in the 83rd percentile are both within one standard deviation of the mean. Most of the large differences in ability, assuming a normal distribution, would be in the bottom 10 percent and top 10 percent. If you are going to punish and reward, it would be best to focus on those teachers and those teachers ONLY, since the shame of the 25th percentile teacher (and the pride of the 75th percentile teacher) is most likely statistically misplaced.

Posted by: jacobh | September 5, 2010 8:00 AM | Report abuse

I don't agreed with cypherp (above) that Leonhardt "does actually have a point."

If you read Leonhardt's piece, he never explains how value-added works. Unless you know how it works coming in, you don't know going out.

I don't follow the point cypherp explains (at all). But is this the point Leonhardt was trying to make? Why would we think so? To all appearances, Leonhardt (who is very smart and a very clear writer) simply doesn't understand the nature of value-added. To all appearances, he was citing the familiar problem value-added was actually designed to address--the fact that Teacher A may be assigned all the "smart" kids, while Teachers B, C and D get assigned less advanced groups of kids.

Posted by: bobsomerby | September 5, 2010 11:20 AM | Report abuse

As a graduate student in Educational Measurement at the School of Education, UC Berkeley, I had to write a paper on the Larry P. decision. A judge had decided that, because IQ tests tended to show lower scores for blacks than whites (PS Asians are the highest), IQ tests could not be used for black students.

I argued this is a different standard than is used in other disciplines. The judge was saying that the risk of a false positive, i.e. calling a student retarded when he/she is not, was the most important risk to be minimized. In medicine, the risk of a false negative is usually the standard - i.e. someone dying when they shouldn't have.

Applying this means we use every imperfect medical procedure in development, but in education - in this Larry P. decision - we are forbidden to use educational procedures while they are developed. Education procedures have to be perfect or not at all. Evidently this argument is still affecting the field.

BTW I asked every single professor I had in 5 years there, "Does teaching spelling teach kids to spell?" The best answer I got was, "Many of our educational practices suffer from a lack of validity."

Posted by: lroberts1 | September 6, 2010 9:41 AM | Report abuse

When parents take the responsibility for their own children instead of relying on someone to raise them, the problems will fall by the wayside. If one can't take care of your own children, don;t have any.

Posted by: Lincoln74 | September 6, 2010 11:29 AM | Report abuse

cypherp said: "A glimpse of these confounding variables can be seen by looking at the effect of 5th grade teachers on 4th grade test scores. Intuition tells us that there should be no effect - events in the future cannot affect the past. However, some value added methods show this effect of future teachers affecting past test scores."

That is a total and complete lie. Correlation is not causation. The future does not effect the past. This is not intuition, it is the nature of causality in the universe, unless you are making a claim about wormholes or something.

On the other hand, it is possible that the 4th grade teacher can effect the student in such a way that they do better in 5th grade. That seems obvious.

Posted by: staticvars | September 6, 2010 12:59 PM | Report abuse

in2liberty:

Awesome plan. You just have to come up with a way to objectively determine who is in that bottom 25%.

Posted by: lol-lol | September 6, 2010 7:11 PM | Report abuse

--"Does anyone give a flying fig about public schools?"--

No, Howler, except the people forced to send their kids to them, and they're powerless to do anything.

Get government out of the education business. It's a disaster.

Posted by: msoja | September 6, 2010 11:23 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company