Network News

X My Profile
View More Activity


Posted at 11:23 AM ET, 08/17/2010

Willingham: Big questions about the LA Times teachers project

By Valerie Strauss

Cognitive scientist Daniel Willingham, a psychology professor at the University of Virginia, takes a hard look at the Los Angeles Times project that evaluated teachers by using test score data. Willingham is the author of “Why Don’t Students Like School?”

After this was initially posted and several readers added their own comments, Willingham responded, with a correction and an explanation. I have included his addition at the end of this post.

By Daniel Willingham
It is dangerous to be right in matters on which established authorities are wrong,” Voltaire said. When it comes to education policy, being right has not looked dangerous in the last few days, but it has certainly looked futile.

In case you’ve been off the grid, the Los Angeles Times hired a researcher to analyze seven years of math and English standardized test scores from the LA school district. Then the Times published an article about the results, profiling some of the “best” and “worst” teachers by name. They’ve also constructed a searchable database that allows one to find the results for any teacher in the district. (Whether the database will be open-access is not clear to me.)

The results are based on a “value-added” measure of teacher performance. The idea, roughly, is that one uses the student’s earlier test score as a control for the later, so that one measures learning (that is, change across time) and not just students’ absolute performance.

The writers of the Times article are either uninformed or disingenuous about the status of the value-added measures. They write: “Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.”

The “others” include most researchers looking into the matter.

Value added models work well when you’re trying to evaluate a school. They work much less well when you’re trying to evaluate an individual class.

Here’s a thought experiment. You and I are equally good fourth grade teachers, but I am more clever about looking professional in front of parents during back to school night. As a consequence, parents think I’m a better teacher. Who will have better value-added measures?

Arguably, I will. If I have a reputation as the better teacher, parents who are more involved in their child’s education will go to the principal and request me. If I have a principal who accedes to such requests, I’ll have a classroom with a higher proportion of kids with supportive, involved parents.

Value added models assume that kids are assigned to classrooms at random. They aren’t.

This is just one problem with value-added models. There are others, which I’ve written about before. Were the writers for the Times unaware of such problems? Were the editors, who put the story on the front page?

I doubt it. I think their reasoning might be revealed in the story’s subheadline: “A Times analysis, using data largely ignored by the LAUSD, looks at which educators help students learn, and which hold them back.“ LAUSD is the Los Angeles Unified School District.

I’m guessing that the editors at the Times are frustrated by the inaction of the LAUSD on teacher evaluation, (or on school quality in general) and they are trying to goad them into doing something.

How should teachers and their unions respond to this?

The head of the Los Angeles teachers' union has called for a boycott of the paper. Righteous indignation is a natural response to injustice, but it’s ineffective and it can slide all too easily into a victim mentality and excessive talk of “what they are doing to us.”

A classmate of mine in graduate school studied negotiation and went to work for the U.S. State Department. He was part of a team working on negotiating water rights between Israel and Jordan.

He was describing the positions of each country in the negotiations when I interrupted him, saying that one of the countries’ bargaining position was predicated on a completely false history of the region.

He said “Dan, you’re thinking that who is right and who is wrong has some bearing on negotiations. It doesn’t. All that matters is each party’s negotiating position and their negotiating strength.”

Teachers and teachers unions would, I believe, do well to bear this in mind.

When it comes to value-added measures, teachers and unions are right. The models aren’t reliable enough to evaluate individual teachers. But right now that doesn’t matter much.

The mood today is that something has to be done about incompetent teachers. We’ve seen that mood in districts in New York City and Washington D.C. and now we’re seeing it in Los Angeles.

We’re also seeing it at the federal level. Education Secretary Arne Duncan said that the publishing of the individual teacher’s scores is just fine.

The people who feel that something must be done are right. In most districts there is not a mechanism by which to ensure that incompetent teachers are not teaching.

I have said before that if teachers didn’t take on the job of evaluating teachers themselves, someone else would do the job for them. The fact that the method is they are using is inadequate is important, and should be pointed out, but it’s not enough.

No one knows better than teachers how to evaluate teachers. This is the time to do more than cry foul. This is the time for the teacher’s unions to make teacher evaluation their top priority. If they don’t, others will.

Now, it may be too late.

-0-

Willingham posted the following in the comments section after a number of readers reacted to the above piece:

As a couple of you pointed out here and others pointed out in private emails to me, I was in error on the random assignment question. I was thinking about the consequences of students not being randomly assigned to classrooms—which is a problem, the seriousness of which is controversial—and I simply didn’t do my homework to make sure that I was right about how that’s treated in the models. I apologize for the error.

I stand by my larger point that VA models are not ready to be used in high-stakes personnel decisions, and I think I’m correct in characterizing that as the opinion of most people who are developing and testing the models.
I didn’t mean for this blog to be about the limitations of VA models, which is why I gave just one example. What I meant to emphasize was that teachers and their unions are unhappy that politicians and now institutions--the LA Times--are imposing evaluation schemes of which they (teachers and union officials) disapprove. But this imposition is a consequence of the profession failing to regulate itself. Indeed, when I hear complaints that teachers don’t get enough respect, that’s one of the things I think of; other professions do a much better job of protecting their own status, and one of the ways they do that is some assurance of quality in its membership. The teaching profession can take that job seriously itself, or its members can watch while someone else does it.

Posted by: DanielTWillingham | August 17, 2010 4:05 PM


Follow my blog all day, every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!

By Valerie Strauss  | August 17, 2010; 11:23 AM ET
Categories:  Daniel Willingham, Guest Bloggers, Teachers  | Tags:  daniel willingham, la times story, la times story on teachers, los angeles times and teachers, research and value added, teacher evaluation, value added evaluation, value added measures  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: The best kind of teacher evaluation
Next: The problem with abolishing the 'D'

Comments

Unfortunately, the thought experiment proposed above was not carried out properly! A teacher would have a biased Value-Added measure only if the parents who chose the showier teacher were differently motivated by the showiness than parents who "chose" the less showy teacher. The Value Added specification controls for fixed student characteristics, including, presumably, a student fixed effect. Of course, a particular incarnation of a Value Added model may not include such student level fixed effects, but that is an implementation issue rather than an inherent shortcoming of Value Added models. There are plenty of those, by the way, other than the one the author suggested.

Posted by: nick_blows | August 17, 2010 11:59 AM | Report abuse

The author pretends to understand how LA Times carried out its value-added analysis, but his thought experiment demonstrates his utter ignorance. It is not a matter of popularity contest and having parents put the smarter kids in the popular teachers class. The LA Times article pointed out how award winning teachers may not be the best teachers according to value-added analysis [e.g. Karen Caruso]. If value-added analysis leads to popularity contests, as the author suggests, Ms. Caruso should have had the best value-added performance, which was not the case. Value-added analysis is about score improvement, not absolute scores. Not that I'm a proponent for value-added analysis, but if the author is going to attack the LA Times piece, he needs to get his analysis/experiment right. You don't get a free pass just because you're a professor. Also, if you're going to put down value-added analysis as an approach, offer a better alternative. It's too easy to criticize.

Posted by: goldenbear1 | August 17, 2010 12:25 PM | Report abuse

Total Quality Management expert W. Edwards Deming (now deceased) pointed out that most organizational problems are structural and not employee based.

Evaluating teachers is essentially counterproductive. It provides no benefit and results in gaming the system. What Deming proposed was empowering workers to make structural changes that improved quality and efficiency. Many educators have proposed the restructuring of education into collaborative learning communities where teachers are required to observe each other and then collaboratively work out improvement strategies. The jargon for this is "capacity building" but it presumes that there are also master teachers assigned to facilitate this and guide the process.

When schools hire teachers they presumably hire the best teachers they can get. Firing your best and hiring worse is not a good strategy. The real problem is the lack of a structural function for organizing the instructional process. I recommend Harvard Professor Richard Elmore's 2002 essay "Bridging the Gap between Standards and Achievement" for an exposition on this.

Posted by: zoniedude | August 17, 2010 12:26 PM | Report abuse

Article referred to in L.A. Times
Who's teaching L.A.'s kids?

"Other studies of the district have found that students' race, wealth, English proficiency or previous achievement level played little role in whether their teacher was effective."
..................................
Yes it is just as likely for a child with very little skills in English to learn reading English from the supposedly average effective teacher, as it is for a child with skills in English to learn reading English, in the same classroom, from the supposedly average effective.

Remember that an average effective teacher is supposed to be solely measured by test results, and according to the L.A. Times, the average effective teacher will always get the test scores of their students up.

How can anyone believe the L.A, Times when it presents such an absurdity that there is no difference in children based upon English proficiency or previous achievement level?

Failing to learn to read in the 4th, 5th, 6th, and 7th grades, according to the L.A. Times, does not make any difference in the ability of a child to learn to read in the 8th grade. This child has the same ability to learn to read in the 8th grade as the child that passed the tests in the 4th, 5th, 6th, and 7th grades and simply requires the average effective teacher. Piece of cake!

The idea that the skills and ability of students have nothing to do at all with the ability of students to learn is an absurdity.

At some point the public will finally recognize that test scores do not measure the effectiveness of teachers but simply measure the ability of students to learn.

Time to recognize that the L.A. Times is simply expecting readers to drink the kool aid.

Posted by: bsallamack | August 17, 2010 1:07 PM | Report abuse

As I understand it, value-added analysis doesn't require random assignment. If the children of motivated parents are more likely to choose Teacher A, then presumably they'll enter with higher scores than the students of Teacher B. If the teachers are equally effective, their students will make equal gains.

I do see problems with judging teachers solely on test score gains in reading and math, but we've got to start somewhere. I don't think teachers' unions are capable of tackling the job themselves, even though many teachers would support it.

Posted by: joannejacobs | August 17, 2010 1:18 PM | Report abuse

The tests given to Los Angeles children are not professionally proctored nor have they been designed to evaluate the effectiveness of teachers.

I believe teachers will win vindication in the courts. In the meantime I've canceled my subscription to the Los Angeles Times because it accepted the test scores as valid without even investigating them, especially in regard to circumstances surrounding their validity and administration. The New York Times is a much better paper anyway so I'd rather read that.

I DO agree that teachers need to be involved in the evaluation of their peers. Teachers know who is effective and who is not. This will come in the near future. I don't believe it is too late.

Posted by: Linda/RetiredTeacher | August 17, 2010 1:21 PM | Report abuse

Unfortunately for this theory, the LA Times article clearly stated that none of the parents had any interest in which teacher their children had. The assignments were as close to random as could be.

Posted by: vinyl1 | August 17, 2010 2:05 PM | Report abuse

Willingham states: "No one knows better than teachers how to evaluate teachers." I'm not convinced that this is true, just as being a great basketball player does not make one a great coach. Teacher evaluation requires a deep understanding of curriculum design, instructional planning, pedagogy, assessment, and classroom management, which not all teachers have. Moreover, many teachers are just not that reflective about their practice. Even good teachers may not understand why they are effective with certain student, nor could they articulate clear criteria for identifying good teaching. I'm not saying teachers should not be involved in evaluation; they should as I think it would benefit them. But instructional leaders, e.g., principals and APs should be trained in teacher evaluation and how to translate it into useful professional development. They should be spending their time in classrooms rather than dealing with discipline, facilities, grant writing, purchasing, etc.

Posted by: gideon4ed | August 17, 2010 2:28 PM | Report abuse

For those interested, a Q and A about our approach can be found here:

http://www.latimes.com/news/local/la-me-qanda-20100816,0,4120439.story

A technical paper on our methodology is
here:

http://www.latimes.com/teachermethod

Jason Felch
Los Angeles Times

Posted by: JFelch | August 17, 2010 2:36 PM | Report abuse

Mr. Felch,

If you are an honest man you will discourage young people from becoming teachers and you encourage all current teachers to leave the teaching field.

You know this is the only honest thing to do.

Posted by: educationlover54 | August 17, 2010 3:18 PM | Report abuse

In regard to the "outing" of the teachers in the Los Angeles Times, let's just consider the case of teacher Karen Caruso, who was publicly labeled as "ineffective" by the Los Angeles Times even though she is considered "well-regarded" by the principal, the parents and her peers!!! It is just incomprehensible to me that a major newspaper would do such a thing and that Arne Duncan would approve it. Are these people conscious? Let's consider the following:

Was this test valid and reliable? That is, was it designed to measure teacher effectiveness? Did most experts concur with the expert consulted by the Los Angeles Times?

Was the test administered under secure conditions designed to protect its integrity? Were there proctors in the room? Did teachers have the opportunity to see the test beforehand and drill their students on the exact items, thereby invalidating the test? Did Ms. Caruso or any of the other teachers coach the children on the test? Did they point to answers during administration? Did they change answers? Did the Los Angeles Times investigate any of this? If Ms. Caruso gave the test correctly but the other teachers did not, how would this affect the test scores of her students?

Are the reporters qualified to give their opinion of a board certified teacher who is deemed "one of the most effective teachers" by the principal. Are these reporters experienced teachers or administrators? Would their testimony be accepted by a court of law?

Did Ms. Caruso know that her scores were lower than those of other teachers? Did she have the opportunity to know how to improve them? Was it ethical for the Times to single her out in this way? Would Jason Felch do this to his mother or his closest friend?

Are there other factors that could have affected Ms. Caruso's scores? For example, is there another equally effective teacher in her building who is more attractive than she is or more "fun?" Could the parents of the more motivated students be requesting the more attractive teacher?

Has Ms. Caruso been libeled? Has she been injured by this article? Is she entitled to damages?

I want to reiterate that I am NOT against publishing the test scores of individual teachers. This is public information and the public DOES have a right to know. However, at the very least, the tests should be valid and reliable, they should be designed to measure teacher effectiveness, the information should be available to teachers first, and there needs to be strict security around their administration, similar to that of the SAT. Teachers need to insist on this.

Never before in my life have I cancelled a subscription to a newspaper, but I did it today. I also informed a solicitor for the Democratic Party that I no longer support President Obama and will not give them any more money.

Jason Felch, shame on you! You must be very young to have participated in this travesty. Please think about how you might have injured many innocent teachers.

Posted by: Linda/RetiredTeacher | August 17, 2010 3:36 PM | Report abuse

As a couple of you pointed out here and others pointed out in private emails to me, I was in error on the random assignment question. I was thinking about the consequences of students not being randomly assigned to classrooms—which is a problem, the seriousness of which is controversial—and I simply didn’t do my homework to make sure that I was right about how that’s treated in the models. I apologize for the error.
I stand by my larger point that VA models are not ready to be used in high-stakes personnel decisions, and I think I’m correct in characterizing that as the opinion of most people who are developing and testing the models.
I didn’t mean for this blog to be about the limitations of VA models, which is why I gave just one example. What I meant to emphasize was that teachers and their unions are unhappy that politicians and now institutions--the LA Times--are imposing evaluation schemes of which they (teachers and union officials) disapprove. But this imposition is a consequence of the profession failing to regulate itself. Indeed, when I hear complaints that teachers don’t get enough respect, that’s one of the things I think of; other professions do a much better job of protecting their own status, and one of the ways they do that is some assurance of quality in its membership. The teaching profession can take that job seriously itself, or its members can watch while someone else does it.

Posted by: DanielTWillingham | August 17, 2010 4:05 PM | Report abuse

"I want to reiterate that I am NOT against publishing the test scores of individual teachers."
_____________________

Okay--Here is the problem. These are not test scores of teachers. They are test scores of students. The teachers have no say in what is on the test, how tested material is worded, how the test is administered etc. The test is designed the achievement of STUDENTS. One thing that complicates this is the fact that there is no incentive for students to perform well on these tests nor is there any consequence for them when they don't perform well. Perhaps tests like the SAT or high school graduation tests would be a better measure. Kids are motivated to do well on those. Perhaps if the tests determined whether or not a student could progress to the next grade, they might be more motivated to do well. I'm not necessarily advocating for this, but the tests simply were not designed to determine how well a teacher has done their job.

It's like saying that a car manufacturer should be paid by how one does passing a driving test. It makes no sense.

Posted by: musiclady | August 17, 2010 4:06 PM | Report abuse

When introducing the teacher unions (NEA or AFT) into education discussions as an equal participant in the development of education policy, the assumption is that all unions are equal...which is the same WRONG assumption we are making about teachers and student learning. We need to break away from generalities if we are truly going to solve what ail's public education.

As a person who was a local President in a state where the "association" is marginally effective and in a district where the association was all but destroyed....that premise of "working together" and the "union" taking the lead is laughable.

I'm not saying it shouldn't happen...but, when the current and past school administration portrayed the quasi-union as the enemy, and the current local President's goal is to be "liked," the chances of a meaningful discussion are nil.

Posted by: ilcn | August 17, 2010 4:07 PM | Report abuse

Musiclady:

Of course you are correct. The test scores considered in the L.A. Times article belong to students, not teachers. This test was not designed to measure the effectiveness of teachers and therein lies the problem. Thanks for the correction.

Mr. Willingham:

Many people don't realize that when teachers first formed an association, they wanted it to be a professional one with professional responsibilities (autonomy regarding curriculum amd instruction, teacher evaluation and induction, etc.) but the legislatures in the various states would only allow them to bargain for "workers rights," i.e. a labor union. Teacher evaluation is a matter of law and will have to be addressed within that context. Unions cannot hire, fire or evaluate teachers. According to law, that is the responsibility of school administrators.

Once during my long career, I was evaluated by other teachers for a state mentor teacher program. That was the most thorough evaluation that I ever received and I also got excellent feedback.

I believe that we will eventually have peer evaluation and review, but it might take a while.

Posted by: Linda/RetiredTeacher | August 17, 2010 4:45 PM | Report abuse

Vinyl 1:

Yes, the L.A. Times reported that none of the teachers was chosen by the parents but how do they know this is true?

I taught for a very long time and I know that these "requests" are often done orally and in secrecy. Many principals go to lengths to keep these requests confidential for obvious reasons. I doubt very much if they'd offer this information to reporters. Did the reporters investigate this or just accept someone's word for it? I never taught in a school where requests were denied.

No group test has yet been developed to evaluate the effectiveness of a teacher. It is just amazing to me that the L.A. Times reporters and Arne Duncan do not know this. A teacher CAN be evaluated but it takes the time and talents of other professionals to do it. Does this really surprise anyone?

Posted by: Linda/RetiredTeacher | August 17, 2010 5:19 PM | Report abuse

The random assignment issue is not essential for doing the VAM calculations, but it is essential if you're trying to draw conclusions comparing one teacher to another. Otherwise, it's like running a lab experiment multiple times without ensuring comparable conditions for the experiment.

The number of factors that affect student performance is unknown, and even it it were known, it would be impossible to control for those factors in calculations.

As for teachers taking ownership of the process, Willingham makes an excellent point. It seems to me that unions have been forced to negotiate in a scarcity model for so long that members can't even envision another way. However, the National Board for Professional Teaching Standards certainly represents a model that could be adapted, expanded, and referred to in these matters.

Posted by: DavidBCohen | August 17, 2010 5:53 PM | Report abuse

@joannejacobs and @goldenbear1

Dr. Willingham is correct – random assignment is a significant problem for most value-added methodologies, including the one apparently used by the LA Times. Although VAM allows test score gains to be isolated, it does not control sufficiently for the fact that different groups of students learn at different rates. In the educational research literature, Stanovich and others have discussed this under the term “Matthew Effect” (see http://www.psychologytoday.com/files/u81/Stanovich__1986_.pdf). Or you might think of it sort of as a kind of mental compound interest – the more you have, the more you earn (or learn). So assuming random assignment of students skews the results.

To use an example different from the one in the article, think about a male teacher who is particularly good at controlling unruly boys. His principal will likely assign him most of the unruly boys at his grade level. All of this teacher’s students might be doing well by the end of the year (with these particular students doing much better that if they were assigned to any other teacher), but his VAM scores are likely to suffer in comparison to the teacher down the hall with a disproportionate share of the well-behaved, studious kids. Thus, rating teachers by this single, narrow, flawed measure is not only unfair, but provides a disincentive for teachers to take on tough assignments.

Posted by: mattmel | August 17, 2010 5:56 PM | Report abuse

Dr. Willingham's example problem was wrong not just because the sample was likely random, but because the article itself shows that neither parents nor principals nor teachers know what traits to select for in order to find a high-performing teacher--so they don't. That's exactly why the data is so important. It can start to slaughter the sacred cows people believe are relevant but in fact aren't.

Sure, there are issues with value added measuring. But to complain before looking at how the analysis was actually done seems a bit smug, to say the least.

Posted by: AllisonLC | August 17, 2010 6:05 PM | Report abuse

--All of this teacher’s students might be doing well by the end of the year (with these particular students doing much better that if they were assigned to any other teacher), but his VAM scores are likely to suffer in comparison to the teacher down the hall with a disproportionate share of the well-behaved, studious kids.

But this is knowable by the data, so the example is poor.

For one, the data would have correlations that aren't present. Second, the data shouldn't show any successful teachers working with the low achieving population, and yet, there are some.

Again, the article looked at the top and bottom decile. If all of these roadblocks are there, how can the top decile perform so well--if all of there students scored so well in the first place, they wouldn't have had the ability to raise their scores so much.

Posted by: AllisonLC | August 17, 2010 6:08 PM | Report abuse

From the LA Times article:
"Third Street Principal Suzie Oh described Caruso as one of her most effective teachers. But seven years of student test scores suggest otherwise."

Caruso, suprised and disappointed by the results: "Still, Caruso said the numbers were important and, like several other teachers interviewed, wondered why she hadn't been shown such data before by anyone in the district."

"For better or worse," she said, "testing and teacher effectiveness are going to be linked.… If my student test scores show I'm an ineffective teacher, I'd like to know what contributes to it. What do I need to do to bring my average up?"

Okay, so Caruso makes a point. By contrasting the two different teachers in the article regarding vocabulary lesson style, one may assume that Nancy Polacheck does indeed demand more from her students and this could be reflected in end-of-the- year test results. Back to Caruso, is it safe to assume that Principal Oh gave Caruso glowing evaluations? Did Principal Oh not see areas for improvement and did Principal Oh not spend time evaluating prior year test results for Caruso (and all the school's teachers) and see any red flags that would indicate that all did not shine so brightly after all?

Okay, so we may be witnessing an evaluation problem and lack of effective educational leadership on the part of Principal Oh whereupon opportunites for professional grown on the part of Caruso were missed due to an inferior observation/evaluation process. Seems that Caruso could have also benefitted from peer observations and some time team teaching/observing as well.

Not a fan of publishing teacher data in this way - and to due so reflects on sour leadership skills from the administrative ranks at high levels.

Posted by: shadwell1 | August 17, 2010 6:11 PM | Report abuse

However, the National Board for Professional Teaching Standards certainly represents a model that could be adapted, expanded, and referred to in these matters.

Posted by: DavidBCohen | August 17, 2010 5:53 PM
______________________
My district's comprehensive evaluation system is based upon the NBPTS model. This is one reason why my district was one of 2 in MD (out of 24) who refused to sign on to the state's RttT application. They felt that their current evaluation model was more comprehensive than what the state was proposing.

Posted by: musiclady | August 17, 2010 6:36 PM | Report abuse

Dan,

"You and I are equally good fourth grade teachers, but I am more clever about looking professional in front of parents during back to school night. As a consequence, parents think I’m a better teacher."

Parents may well think you're a "better" teacher because you put on a better dog and pony show at open house but this is a subjective assessment (on the part of the parents). There is no objectivity involved in this form of an evaluation.

The primary rationale of employing value-added measures in evaluating teachers is to inject (at least) a degree of objectivity into the process. I've stated this on numerous occasions: the existing system of administrative subjective evaluations is an embarrassment to the teaching profession. IT MUST BE CHANGED. Parents and taxpayers have a right to know; which teachers are, and which teachers are not effective in the classroom.

As well, districts that are headed in this direction (Washington DC) are using student test scores as part of a "mixed measures" approach, added to principal and peer evaluations. The value appropriated to student test scores has been negotiated through the collective bargaining agreement, not simply rammed down the teachers' throats.

Additionally, as I've also stated on numerous occasions; what kind of a teacher would be reluctant to be evaluated on the performance of their class? To be eligible for additional compensation outside the anachronistic salary schedule? Anyone care to hazard a guess on that one?

Posted by: phoss1 | August 17, 2010 6:47 PM | Report abuse

I liked Shadwell's points. Principal O was negligent on her evaluations. Why did Caruso have a history of favorable evaluations? Was she buddy/buddy with the principal? Was she a good BS artist. Also, good for Caruso for wanting to improve her craft and find out where she could have been deficient. This kind of process is vital to improving the way we evaluate teachers.

And BTW, teachers are NOT the problem, as anyone who has ever been in the classroom knows. Teachers can be conditioned into at least a degree of effectiveness through professional development, mentoring, etc. No, the problem is the kids who show up a the school house gate, or rather the parents who screwed these kids up over the first five to six years of their lives. That's the problem, not ineffective teachers.

Posted by: phoss1 | August 17, 2010 7:00 PM | Report abuse

Additionally, as I've also stated on numerous occasions; what kind of a teacher would be reluctant to be evaluated on the performance of their class? To be eligible for additional compensation outside the anachronistic salary schedule? Anyone care to hazard a guess on that one?

Posted by: phoss1

Quite a number of Catholic school nuns didn't need "additional compensation" to teach their classes of 50.

Posted by: phillipmarlowe | August 17, 2010 7:02 PM | Report abuse

Valerie and Daniel,

Thank you for always telling the truth. I appreciate you more than I can say.

Posted by: educationlover54 | August 17, 2010 7:54 PM | Report abuse

Funny thing about testing....we all have our off days, off hours, off minutes. Take for instance my post in which I end the sentence with "due" rather than "do" and didn't catch it before hitting the submit button. Weird. And how did I even type the wrong word/homophone? Who knows, but thanks for not answering anyway. (Actually, I posted in somewhat of a hurry as I was about to prepare dinner and also awaiting a call from my daughter about a car repair.) Now, realizing that if a kid is feeling badly the day of a test, his head really isn't in it, or whatever, what kind of stuff will slip by that wouldn't if tested on the next day? A lot of pressure on kids, and now teachers, bearing sometimes unreliable results.

phoss1 - thanks.

Posted by: shadwell1 | August 17, 2010 7:58 PM | Report abuse

phillipmarlowe:
"Quite a number of Catholic school nuns didn't need 'additional compensation' to teach their classes of 50."
____________________________

The nuns may not have needed additional compensation to teach 50 students, but they had the use of rulers to strike students and the full backing of parents.

Posted by: PLMichaelsArtist-at-Large | August 17, 2010 8:04 PM | Report abuse

For those interested, a Q and A about our approach can be found here:
http://www.latimes.com/news/local/la-me-qanda-20100816,0,4120439.story
A technical paper on our methodology is
here:
http://www.latimes.com/teachermethod
Jason Felch
Los Angeles Times
Posted by: JFelch
.....................................
The usual with the pretense that if we could quantify exactly every factor that goes into learning we could predict the test scores of students given the mythical Average effective teachers.

Sell this to the farmer where if you could quantify every factor that goes into growing a crop for a year you could predict the crop yield for the Average effective farmer.

Of course the authors of the paper are not going to quantify every factor and pretend a great number of the factors are not important and can be simply set to zero. Zero for innate ability of the child and zero for the composition of the class that the child is in.

One tires of these frauds.

Yes if every circumstance of every child entering grade 4 was the same you could predict their scores with a mythical Average teacher. The problem is that every child is different and you can not even measure the differences.

Remember the great mathematical models that predicted for investors how much profit they would make from bundles of mortgages that suddenly turned worthless.

DO NOT BOTHER TO VISIT THE WEBSITES.

They lead to the mathematical paper "How Effective Are Los Angeles Elementary Teachers and Schools?", where you find that that this model simply disregarded factors such as the composition of classes and the innate ability of children.

"The model was simplified by assuming that the student heterogeneity term (αi) was zero. This assumption was consistent with initial data runs that indicated that student heterogeneity was statistically insignificant after controlling for prior year test score and observed student characteristics."
....................
By the way this supposedly mathematical paper is not very well written. Plenty of greek letters but no indication of the actual value that will be used for the formulas.
Yes I can can predict the stock market value a year from now and here is my simplified formula with greek letters but I will not tell you which set of values to use for the the various factors. Just send the check and you will be rich!

For someone who once worked in Wall Street this "mathematical" paper would be rated an F.

Posted by: bsallamack | August 17, 2010 8:24 PM | Report abuse

I just tried to post the following comment on the LA Times and couldn't get it to work:

"Ms. Caruso was an amazing teacher," said Rita Gasparetti, whose daughter was in Caruso's class a few years ago. "She really worked with Clara, socially and academically."


Perhaps Ms. Caruso was considered 'an amazing teacher' because she taught the students other things besides the data the tests measured......the mother says that Ms. Caruso 'really worked with Clara, socially.....' so maybe the socialization the daughter got in Ms. Caruso's class helped her get along and do well in other classes, yet that socialization process is not being measured.

Chances are, if parents are saying good things about a teacher, he/she is doing quite a few other things right,and doesn't deserve to be brought to national attention and chastised in the manner this newspaper [LA Times] is presenting.


Posted by: PLMichaelsArtist-at-Large | August 17, 2010 8:36 PM | Report abuse

Additionally, as I've also stated on numerous occasions; what kind of a teacher would be reluctant to be evaluated on the performance of their class? To be eligible for additional compensation outside the anachronistic salary schedule? Anyone care to hazard a guess on that one?

Posted by: phoss1
..........
Posted by: phillipmarlowe
.............................
A smart teacher is the answer to the repeated question of PHOSS1.

What kind of American would be reluctant to play at the gambling casino and be eligible for additional compensation outside the anachronistic salary schedule?

A smart American.

phillipmarlowe do not bother with PHOSS1.

PHOSS1 would be squealing like a stuffed pig if his boss told him he would make a few bucks if he got a number up that was beyond his control, or be fired.

I have answered this question before for PHOSS1.

PHOSS1 is like some of the really unintelligent employees I have had in the past working for me. I had to constantly follow them around and answer their stupid questions over and over.

PHOSS1 the next time you destroy data in the computer system again, I will fire you so fast your head will spin.

By the way when I worked in Wall Street there were 5 bonuses a year.

Any employer on Wall Street that told me that my job was dependent upon things out of my control, I would considered to be an idiot and not worth working for.

Any teacher or union of teachers that accepts the idea of being fired on the basis of the test results of their students should have their head examined.

Posted by: bsallamack | August 17, 2010 9:09 PM | Report abuse

I love Americans.

Use the term "value added" and immediately Americans assume it means something.

How about "value added" for the Emperor's new clothes?

The "value added" is the savings in cloth and thread.

Then there is the "value added" in buying stock A instead of buying B.

Great to know you bought a "value added" stock when it tanks.

Use the term "value added" and Americans will believe anything.

Posted by: bsallamack | August 17, 2010 9:21 PM | Report abuse

I'm pretty sure the journalists who wrote this article, like most journalists, were more interested in getting a big story than in anything else. Their names are really up in lights right now. Tons of Google hits. What an adrenaline rush.

Secondarily, they may think (and they may or may not be right) that they have done a GOOD THING for education by publicly exposing bad teachers and possibly changing the face of education by their brilliant, ground-breaking investigative reporting.

They may be very smart, but they may be even more arrogant and ego-driven and heartless.

I predict that soon, they will be making excuses for themselves and will be feeling the rightful wrath of teachers and students and parents, and will regret the hubris that led to this article.

Posted by: efavorite | August 17, 2010 9:39 PM | Report abuse

Congratulations to Ms. Caruso for being thoughtful about her teaching, and what she might try to improve. That's the right attitude, and a far cry from the many ostriches posting here who want to do everything they can to avoid using data.

A different study released by the University of Virginia showed that teachers could create highly socially effective classrooms -- some of which were highly academically effective, and some of which weren't. While many parents (and Ms. Caruso's principal) may have applauded her for the stimulating and socially effective classroom environment, it is also important that students make significant academic gains, particularly in the core subjects of reading and math. A teacher whose students make poor gains has some significant opportunity to improve, and I am sure that someone of Ms. Caruso's skill can do so -- once she has the information that she needs to improve.

The shame is LAUSD's, for failing to make use of this kind of information years ago. We need more effective teaching, and hiding results makes it much harder to make real improvements. Rather than wailing about the indignity of it all, why not spend some real time trying to understand what the high value added teachers are doing, and try to replicate that.

Posted by: bk0512 | August 17, 2010 9:42 PM | Report abuse

There seems to be a misunderstanding about the "professional" label for teachers.

From my perspective, teachers will not be able to create a system to maintain quality control in the profession since state laws basically make it impossible. There isn't a requirement to have 100 engineers in each of the worst inner-city neighborhoods, or 50 lawyers in each small Appalachian town.

Teachers are only treated as "professionals" in one way: no overtime.

Posted by: someguy100 | August 17, 2010 10:04 PM | Report abuse

Teachers are only treated as "professionals" in one way: no overtime.

Posted by: someguy100
************

And yet, teachers put in overtime because they want to do a good job and they want their students to learn and they know that there's more to a person that a score on a test. (They also put out their own money.)

There's that line from the 1960s show, The Prisoner, that deserves being printed:

" I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered. My life is my own."

Posted by: edlharris | August 17, 2010 10:30 PM | Report abuse

I hope you all are commenting on the LA Times. The readers need to know that the article was biased.

Posted by: educationlover54 | August 17, 2010 10:32 PM | Report abuse

bk0512:

If you read the posts again, you will see that many of us do not believe that "this kind of information" can tell us who is effective as a teacher and who is not. The tests the students took were not designed to evaluate teachers and they were not administered to protect the integrity of the test. Imagine if the Advanced Placement tests were delivered to the classroom a week in advance and then administered by the classroom teacher!! I am simply amazed that so many people are accepting these tests scores as valid!

Several people have asked for an alternative to evaluating teachers. In my opinion, it's not that difficult. An administrator or team of teachers could visit a teacher's class several times a year to observe lessons and to review student work. In addition to this, benchmark tests can be administered to the students by the teacher or principal with proctors in the room. Also, as most teachers know, student growth (or lack of it) becomes obvious by examing their compositions throughout the year. If standardized tests are to be used, they'd have to be different each year (duh!) and administered under strict conditions. If these tests are used to evaluate teachers, they'd have to be designed to do that.

As a reading specialist and mentor teacher, I went into many classrooms throughout the year. It's not that difficult to evaluate a teacher but it takes time and requires the input of other professionals. At this time, there is no single test that can accomplish this task. Or, put another way, there is no cheap and easy way to evaluate a teacher, or any other professional.

Posted by: Linda/RetiredTeacher | August 17, 2010 10:34 PM | Report abuse

Let us admit it. There is something wrong with teachers.

Why? Because they have accepted the kool aid about "effective teachers".

We are losing the war in Afghanistan? No soldier believes that it is because we are lacking effective soldiers.

Crime is high in certain neighborhoods. No police officer believes that it is because we are lacking effective police officers.

Patients die in hospitals. No doctor or nurse believes we are lacking effective doctors and nurses.

Children have difficulty in learning and the reality is that this is not because we are lacking effective teachers.

Time for the teachers to stop drinking the kool aid and tell their unions not to accept this lie.

Band together and tell every neighbor or parent to vote against every politicians or political leaders who are attempting to blame the problems of public education on teachers.

Teachers should email their members of Congress and flood the Obama website with rage at being made the scapegoats for the problems of public education by politicians.

Go to political meetings with signs proclaiming signs to stop blaming the teachers.

And stop posting comments in regard to nonsense about effective teachers. There are problems in public education and it is not because of effective teachers, but the politicians that want to simply place the blame on teachers.

Posted by: bsallamack | August 18, 2010 12:14 AM | Report abuse

bsallamack: you're right, you're right, you're right......

And this political nonsense is keeping us from dealing with much more serious issues in eduction.

Posted by: PLMichaelsArtist-at-Large | August 18, 2010 12:55 AM | Report abuse

This will teach me NOT to post late at night:

correct spelling of: education

Posted by: PLMichaelsArtist-at-Large | August 18, 2010 12:57 AM | Report abuse

Post the names of the testing profiteers, corporations, contractors, subcontractors, executives, politicians, and lobbyists at the high-stakes feeding trough. Publish the multi-billion dollar contracts between the testing conglomerates and all state education agencies. Expose the university professors who receive kickbacks from the testing industry. Investigate the manipulation of high-stakes cut scores to benefit political insiders. The corporate testing industry is like Enron and Wall Street. Millions of teachers and students are used for test-based profit-making without educational "value."

Posted by: nfsbrrpkk | August 18, 2010 7:52 AM | Report abuse

nfsbrrpkk: I am with you......and don't forget the contracts with technology vendors.

bsallamack: "Teachers should email their members of Congress and flood the Obama website with rage at being made the scapegoats for the problems of public education by politicians." Yes...It is past time for this to occur.

Posted by: ilcn | August 18, 2010 9:45 AM | Report abuse

Posted by musiclady: One thing that complicates this is the fact that there is no incentive for students to perform well on these tests nor is there any consequence for them when they don't perform well.

The students are more than aware of this and you would not believe the number of students who tell me that they just filled in whatever mostly because they are sick and tired of standardized testing (exit exams, PSAT, SAT, ACT, Benchmark tests, etc.) and they know this one does nothing to them.

If someone knows how or why this test has become the accepted "performance" upon which my life's work and dedication will be judged by, please explain it to me because I feel lost when trying to understand this.

Another question I have is this: Every year there are cuts made to the budget resulting in less professional development, more students in the classroom, deteriorating buildings, less materials, furlough days where students aren't even there, extra work because the support staff has been laid off. How are these factors accounted for when you are tracking the same children progressing through an increasingly deteriorating environment?

Posted by: lateacher | August 18, 2010 12:31 PM | Report abuse

I'm an adult/HS teacher in the LAUSD and I haven't been this hopeful about positive change since Obama ran for president. I hope this turns out to be less of a letdown than that.

Posted by: LAtchr | August 18, 2010 1:03 PM | Report abuse

A couple things:

1) I'm all in favor of assessing teachers by their students' performance.

2) Note that this was done in elementary school. Middle and high school would have to use very different methods, as the students start almost every course from scratch (except English). Ideally, students would be tested at the beginning and end of the course, and the teacher rated on how the students moved up or down the national (or state) percentile. But even without that caveat, it'd be good to see this sort of testing.

Wouldn't it be nice if there were a few basic standards in place? For example, teacher isn't assessed on absent students, teacher can remove students from the class as needed, students aren't moved forward unless they scored "basic" on the last subject's proficiency test?

Oh, wait. I wrote that already!

http://www.washingtonpost.com/wp-dyn/content/article/2010/06/17/AR2010061704565.html

Posted by: Cal_Lanier | August 18, 2010 1:28 PM | Report abuse

PLMichaelsArtist-at-Large,

Try again, otherwise I will post it for you.

We have got to fight this.

Posted by: educationlover54 | August 18, 2010 2:16 PM | Report abuse

Teachers did take the lead on this many years ago. We established the National Board for Professional Teaching Standards, which offers certification BY teachers FOR teachers.

Posted by: aliceannaa | August 18, 2010 6:52 PM | Report abuse

to educationlover54:

Thanks for the encouragement; I just tried again and got the comment re Ms. Caruso posted in the LA times.

Posted by: PLMichaelsArtist-at-Large | August 18, 2010 8:41 PM | Report abuse

Any employer on Wall Street that told me that my job was dependent upon things out of my control, I would considered to be an idiot and not worth working for.

Ballsamack, what world are you living in? Everyday people lose their jobs because someone higher up has been stealing money, is totally incompetent and runs the business into the ground, decides to save money by firing x number of people no matter what they do, or, in at least one case that I personally know of, buys another company without even knowing what it does.

Posted by: sideswiththekids | August 19, 2010 6:22 PM | Report abuse

PLMichaelsArtist-at-Large,

So glad you got it posted, poor Ms. Caruso - I wish her the best.

sideswithkids - When I worked in business I saw so much insanity at times, it was like working in an insane asylum. Incompetence was rewarded if you were a friend of the boss. Raises were based on who you knew, not what you knew. And management kept firing each other. Come to think of this, this is an environment much like the one Rhee has created in DC schools.

Posted by: educationlover54 | August 19, 2010 7:55 PM | Report abuse

"Remember the great mathematical models that predicted for investors how much profit they would make from bundles of mortgages that suddenly turned worthless."

But Bsallamack, businesses are perfect in every way! That's why we have to model our schools after them. Remember, that great fount of wisdom Blarney Duncan told us so.

Posted by: educationlover54 | August 19, 2010 7:57 PM | Report abuse

I am a parent of several elementary school children, living in California. Our elementary school was recently able to rid itself of a teacher that was much unliked by both parents and students. Why was it so hard to get rid of that teacher?

Parents realize how important education is. We are starved for information on how to give our children the best opportunities. All we have now are API scores (which I understand are more a reflection of socioeconomics). Value added scoring of teachers is just one more bit of information we might be able to use to the benefit our children.

We judge doctors by how many lawsuits they've had (just go to the Medical Board of California website). Is this naming and shaming? No, we feel that we have a right to know if a doctor is a "bad doc", and we are using a poor measure (number of lawsuits) to determine this. Why should the teaching profession be held to a lower standard? I am more concerned with my children's teachers than my own physician.

Finally, the method used by the LA Times is only one method for evaluating teachers. We should be doing random student assignment to teachers, and be doing within-school comparisons of teachers at the same grade level. In this case, the statistics are simpler and more certain. We might even be able to use non-parametric statistics to infer that one teacher is less effective than another. To my knowledge, this is not being done in any school district in California. This must change, and the various teacher's unions should all publicly state that their members are willing to monetarily rewarded and punished (as I am in my profession) if robust non-parametric statistics show that they are an effective or ineffective teacher. (I am losing 5% of my pay this year because my coworkers and I averaged more than 1 accounting error per month).

Posted by: cypherp | August 23, 2010 10:55 PM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company