Network News

X My Profile
View More Activity

Stakes and mistakes in assessing teacher effectiveness

Ms. Mathews and I are off in France, where I haven't been in 45 years and she has never been. I have left some blog posts behind that will pop up while I am away. That includes this guest column by Robert C. Pianta. He is dean, and Novartis U.S. Foundation Professor of Education, at the Curry School of Education at the University of Virginia.

He is one of America's smartest educational scholars, with a long history of observing kids and teachers in classrooms. His work focuses on the assessment of teacher quality, and policy and practice that enhance children’s outcomes, school readiness, and later achievement. Here, he addresses the question of the year: What is the best way to figure out how well a teacher is doing?

Teacher evaluation is emerging as the central flash point in education policy debates. The recent controversy in Los Angeles over publication of teachers’ student test score gains illustrates this. So does D.C. Mayor Adrian Fenty’s reelection loss following his school chancellor’s firing of 173 teachers who were rated “ineffective.”

Both incidents drew national attention because they exemplify an approach to teacher effectiveness aggressively promoted by President Obama and Secretary of Education Arne Duncan -- both rhetorically and in the Race to the Top and I-3 grant programs. Teacher evaluation was the main focus of NBC’s “Education Nation” coverage; one segment featured New Jersey Governor Chris Christie ranting over teacher unions’ defensive stance on evaluation.

Teacher evaluation is controversial because it combines two elements new to education professionals and the public – quantifiable measurement of performance, and stakes like firing or public exposure. Teachers matter. But the core problem in public education is not identifying effective teachers. It’s that our existing system does not produce effective teaching in sufficient scope, scale, regularity, or intensity.

Student test scores, and whether we post them, pay for them, fire teachers, or reward kids, are simply not in themselves a sustainable solution for the lack of effective teaching. Consider the Los Angeles example: How is posting a teacher’s student test scores supposed to lead to better teachers?

Using only test scores as a way to improve teaching is like telling an orthopedic surgeon performing a hip replacement, “Good job, the patient lived” or “bad job, the patient died.” Mortality is important to the surgeon, just as student achievement is important to a teacher. But it carries virtually no information on how to do hip replacement surgery any better the next time.

It’s the same with test scores and how to teach reading to 30 seven-year-olds. Our nation, and particularly its educators and children, are desperate for better information on the techniques teachers can use to foster achievement, problem-solving, getting along with other kids, and a host of other things we value. We even more desperately need ways to produce those behaviors in the 350,000 new teachers who enter the profession each year and the millions already in classrooms.

There’s hope on both fronts. Over the last ten years, at the University of Virginia, we have developed observational assessments of teachers’ behaviors in classrooms. These produce quantitative measures of teaching quality in the classroom. We have found direct links between the extent to which teachers implement certain types of behaviors that we can observe and the resulting student learning gains, motivation, engagement, and social development.

These observational methods are not just one person’s view of effective teaching. We approached observation of teachers’ behavior with the same scientific rigor used to develop measures of student achievement, so these quantitative measures provide robust estimates that regularly forecast student learning.

More importantly, we have shown that if you use a standardized observation of what teachers do that matters, it is then possible to engineer supports to target and improve those behaviors. Not surprisingly, we and others taking this approach have found that targeted systems of professional development that focus on teachers’ observed classroom behavior produce effective teaching and improvements in student learning.

Nor are these observational measures and associated supports too difficult or expensive to implement at scale. Unlike nearly every other method for observing teachers' behavior and interactions in the classroom, this approach has been reliably replicated in tens of thousands of classrooms. The work in these classrooms shows that scalable, standardized measures of teaching behaviors meet the dual challenge of both assessment and improvement.

So what’s the take-away for school districts or reformers? First, if you want to not only measure teacher performance, but also build a system that improves classroom teaching directly and systematically, observational measures are a good bet.

Second, don’t think that local, homegrown, observational “rubrics” or measures mashed-up from an assortment of approaches that look good to a committee will do the trick. If we are going to get traction on the core problem, then individual teachers, schools, and districts must all use quantifiable, non-subjective, proven-to-work measures. There is no shortcut. Local, bottom-up construction of such measures wastes time and is likely to fail.

Third, connect the observational metrics to both carrots and sticks. This includes sanctions for teachers who interact so poorly with students that they actually contribute to student disengagement and failure. But also link them to incentives, rewards, and most importantly, targeted supports to help teachers produce effective behaviors in the classroom.

Test scores alone won’t produce effective teaching, and neither will fuzzy observations and coaching. In the end there is no substitute for rigorous, standardized measures and proven-effective approaches to improve teachers’ instructional behaviors. We already rely too much on luck to produce an effective classroom teacher. Let’s not keep repeating that mistake in our efforts toward reform.

Read Jay's blog every day, and follow all of The Post's Education coverage on Twitter, Facebook and our Education Web page.

By Jay Mathews  | October 24, 2010; 6:00 PM ET
Categories:  Jay on the Web  | Tags:  Robert Pianto, University of Virginia, how best to assess teachers, research on teacher observation  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Students reject their school's high rank

Comments

Dr. Pianta is an actual researcher and scholar, who has done worthy work on the aquisition of mathematical knowledge. But who is he to demand sole authority over the assessment of good teaching? His demand for centralized authority to dismiss all independent development of teacher assessment is based on this weak assertion:
"Second, I don’t think that local, homegrown, observational “rubrics” or measures mashed-up from an assortment of approaches that look good to a committee will do the trick."

But then Dr. Pianta offers his system as one component of the Gates plan for value-added method (VAM) test-score analysis, to which we all must bow. I guess he’s willing to let Gates mash together whatever he wants, as long as it isn’t “home grown”.

Gates might not have expected VAM 7.0 would be so discredited in 2009, when he brought Pianta onboard with a big grant. Gates hired Dr. Pianta to provide cover for its efforts to force the' flawed ad expensive "value-added" metric on public institutions across the United States.

Here is Steve Sawchuck’s September 2009 report, in Edweek;

“The research agenda makes up a component of the foundation’s five-year, $500 million push to define and promote effective teaching practices, and will help shape its approach to the teacher-effectiveness plans it intends to fund this fall.”

“Gates officials could not provide an estimate of the overall amount they will spend on the research, but say the figure will be in the tens of millions of dollars. New York City will receive a $2.6 million grant to participate, while Charlotte is poised to receive $1.4 million”

“Digital videos of teachers’ practices will be a central feature of the research.... and the Classroom Assessment Scoring System, developed by Robert C. Pianta, the dean of the education school at the University of Virginia, in Charlottesville.”

http://www.edweek.org/ew/articles/2009/09/02/03gates.h29.html

Gates strategy is to use Pianta to play good cop/bad cop as he pursues his demand for absolute control of teaching in every school in the country. We are supposed to embrace some mishmash of VAM that includes Pianta, because the alternative is undiluted stupidity. Mathematica Institute's already famous analysis of value-added methods (VAM) has completely debunked any claim to scientific validity for his agenda, however. We don’t have to embrace any of it.

Gates is famous for wresting control of a market by sowing "fear, uncertainty, and doubt" among all other stakeholders. Seriously, Google “FUD Gates” if you want to read about his business practices. VAM is a perfect tool for the FUD master. It is a random-number-driven attack machine, useful only to create a climate of fear, uncertainty, and doubt across the institution Gates aims to dominate: American public education.

Posted by: mport84 | October 24, 2010 10:37 PM | Report abuse

You can purchase Dr. Pianta's Classroom Assessment Scoring System™ (CLASS™) at
http://www.brookespublishing.com/store/books/pianta-class/index.htm
The PreK-K volume costs $50 as does the K-3 volume.

"When it comes to what makes a great teacher, credentials are nice, but don't overlook the importance of positive early childhood teacher–student interactions—the primary ingredient in creating quality educational experiences that launch future school success. With CLASS, educators finally have an observational tool to assess classroom quality in pre-kindergarten through grade 3 based on teacher–student interactions in the classroom rather than evaluation of the physical environment or a specific curriculum. "

Posted by: edlharris | October 25, 2010 12:41 AM | Report abuse

It seems that the U.S.A educational system can not escaped from or obsessed by the topic of test-score either. Yes,education assessment is really a complex work, for it involving teacher and students, both of them are living factors, seasoned by textbooks, national curriculum criterion embodied in it, while the ultimate goal of education, the development of students pointing to future, while both the students and the society in which the students will have some role to play are all in the process of shaping. In one word, in the problem of educational assessment we should attempt to reach an equilibrium between the students present performance and its potentiality in the future development, here another problem might be asked naturally is the score of student's score represent everything?

Posted by: fortune1002003 | October 25, 2010 8:49 AM | Report abuse

The outcome of the k12 education process, which is much broader than just the time spent in school depends on 3 elements:

1. The student;
2. His or her parents (family) + the environment);
3. The school system, of which the teacher (presently found guilty of everything) is just a part.

All three should be held accountable, not only "the teacher" ! If students systematically misbehaves in school ("Waiting for Superman" did not show this, did it ?) don't say: they have rights, we cannot punish them ! All students have rights, including those who ***want*** to learn and are prevented from doing so by those who do not !

Accountability and responsibility from students and their parents as well !

Nobody (especially politicians) has the courage to speak about this ! That's how it's done in those other country featured on American websites and TV: Finland, Singapore, South Korea, etc !

Posted by: dreyfuss1 | October 25, 2010 9:51 AM | Report abuse

Sounds a lot like the assessments under the old Virginia BTAP program. How is your proposal different? What supports will you put in place to help teachers learn the correct "observable behaviors" that will help students achieve? Is this just a construct to help justify more pedagogy courses in an education degree, or is there a way to screen FOR these behaviors, so that more potentially successful teachers could be recruited from other professions? THAT, to me, would be a great solution.

Posted by: diana_prince | October 25, 2010 10:08 AM | Report abuse

Two issues.
Do the people who praise the effectiveness of the rating system have a vested interested in the success of the system.
Has the system really been evaluated in tens of thousands of classrooms?
Descriptions of the application of teacher rating methodology which have appeared in the Washington Post over the past three years do not support its meaningfulness.
John Dickert
Mount Vernon Farms

Posted by: 12191946 | October 25, 2010 11:13 AM | Report abuse

Dean Pianta says the following:

..."we have developed observational assessments of teachers’ behaviors in classrooms. These produce quantitative measures of teaching quality in the classroom. We have found direct links between the extent to which teachers implement certain types of behaviors that we can observe and the resulting student learning gains, motivation, engagement, and social development."

But Pianta never identifies any of those "behaviors."
If these "behaviors" are supported by scientific rigor, Pianta should tell us what they are.

More importantly, Pianta doesn't say anything at all about the socioeconomic connections to school achievement and learning.
His argument for school reform dovetails with that of the U.S. Chamber of Commerce in that "effective education" is defined as a uniform set of behaviors that produce higher standardized test scores. ...and those test scores are critical to the "economic competitiveness" of corporations (of course, the Chamber thinks more tax cuts and more deregulation are even more important, even if they lead to higher deficits, more off-shoring of jobs, more economic dislocation, and increased poverty).

Pianta offers up a behavioristic model of reform, based on "standardized observation" coupled with "carrots and sticks."
The "carrots" include "incentives" and "rewards," and the "sticks" include "sanctions." But behaviorism has a very poor track record improving internal motivation and morale: imposing an externally developed evaluation on people who've had little to do with its development and implementation is a recipe for failure.

The plan Pianta envisions sounds exactly like what's transpired in the District school under MIchelle Rhee. As others have documented, the best gain scores in the District are directly attributable to Clifford Janey (Rhee's predecessor) and not to Rhee-form. In fact, there have been diminished gains under Rhee and even some declines.

And this model is what Pianta thinks others should copy?

Maybe Pianta is not nearly as smart as Jay Mathews thinks he is.

Posted by: DrDemocracy | October 25, 2010 11:51 AM | Report abuse

Thanks mport84. I can always count on the commenters to provide the thoughtful analysis and research I used to see in the print edition of the Post.

Posted by: Title1SoccerMom | October 25, 2010 12:10 PM | Report abuse

"But the core problem in public education is not identifying effective teachers."

No. That is not the CORE problem in public education. Sure, there are some ineffective teachers that should not be in the classroom, but there are others that need supportive assistance to become great teachers. Attitude, seeking assistance, guidance, watching master teachers in the classroom, workshops, etc. are quite helpful in addition to effective principal evaluations in the classroom.

Teaching is part art and part science and all human. Teachers are not automated action figures needing standardized format of observation mandated from, where is that, oh, right, not any "homegrown" local district or fantastic principal??? With all due respect to UVA, their noted standardized evaluation method is A method, not THE only effective method on earth.

I also want a doctor, PA, or NP who doesn't rely on a auto-diagnosis gizzmo evaluation from a computer program. We have insurance companies trying to run healthcare and now all sorts of standarized teacher evaluations gurus trying to run public education. $$$$$$ No thanks.

And, yes, mport84, thanks. Nice work.

Posted by: shadwell1 | October 25, 2010 12:44 PM | Report abuse

This experience that I had the other day should give us direction as to where we need to go:

I was at a party attended by many of my teacher friends who are all still working. There were about twenty teachers, all women and all over the age of 55. Within the next five or ten years, all will probably be retired. These women came from a generation where their only career options were nurse, secretary or teacher. I can't speak for them, but for me engineering was something the boys did. I don't think I even knew what it was.

None of the sons of these teachers is a teacher himself or preparing to be one, but that's nothing new. What is new is the fact that not a single daughter or daughter-in-law are K-12 teachers. They are physicians, attorneys, event planners, college professors, nurse practioners and so forth. The most telling comment came from one teacher who said, "Kristen has her master's and is looking for a community college job. She knows not to go into public school teaching."

Can we see the writing on the wall here? As soon as the economy picks up and the baby boomer women are retired, who will be there to teach the children? I predict a teacher shortage the likes of which we've never seen before.

Teachers are extremely bitter about the bashing that is going on at the present time. They are telling their children, students, relatives, friends and neighbors to think twice before choosing teaching as a profession. It appears that many young people are listening.

In the future, how will we encourage talented college students to consider teaching as a career and to stay with it for more than a few years? That's the big problem. My suggestion is to strengthen the teaching profession so that teachers are in charge of schools in the same way that physicians are in charge of clinics and attorneys are in charge of law firms. Intelligent people want to be decision-makeers. They will refuse to follow a script.

Forget the "sanctions." If things go as I think they will, we'll be back to hiring anyone "with a warm body." Michelle Rhee hired inexperienced people right out of college even though she had a golden opportunity to hire experienced teachers with proven track records of success. I'm not certain of the reason for this but I think it might be spelled M-O-N-E-Y. Nothing has hurt American education more than this shameful practice.

If we want a quality education for our children we have to hire the best teachers possible and then treat those people like the treasures that they are.

There are no shortcuts to a good education and it isn't cheap.

Posted by: Linda/RetiredTeacher | October 25, 2010 12:59 PM | Report abuse

Thank you, mport84. Your words, as well as this article by Dr. Pianta, reinforce my strong belief that the education "reform" movement is actually about money. (e.g.Don't use local systems of evaluation; they won't work. Use the one I've developed so I can make billions of dollars and I won't have to teach a single child, either.) Siphoning off school tax money for personal gain is the absolute last thing we need in public education.

Posted by: Linda/RetiredTeacher | October 25, 2010 1:38 PM | Report abuse

I wonder what type of teacher behaviors he thinks are efficient. Do the behaviors help teachers handle student with behavioral problems.

I wished he had given an example of the behaviors.

Posted by: jlp19 | October 25, 2010 4:59 PM | Report abuse

Some of the usual commenting onlookers are true to form: only comfortable if there is a purported plot by outsiders, and, if there is a good chance the adopters of someone's new or different idea can lead to victimhood, or just playing the victim.

And Linda, bless her heart, believes this is all about money.

This is all part of the we-won't-change consortium. Yup, we got a real success going here. Just look at the results.

Posted by: axolotl | October 25, 2010 5:28 PM | Report abuse

I will be more inclined to accept this latest revelation once it is first applied to Fortune 500 corporations. Once Wall Street embraces the observational method as a compensation tool, so will I. Until then, I ask your readers to enter the real world and visit my blog at teachermandc.com.

Posted by: dcproud1 | October 25, 2010 7:18 PM | Report abuse

dcproud1 = there is nothing to force or link Fortune 500 practices with those applied to public employees. Wake up; this is America, not the Soviet Union of old. You will wait forever. The voters/citizens/taxpayers will determine what is required of public employees. Not the unions. And don't try to ignore the contract overwhelmingly approved by teachers.

Posted by: axolotl | October 25, 2010 8:09 PM | Report abuse

With philanthropic money and total backing from the mayor, Michelle Rhee had a once-in-a-million opportunity to recruit the very best teachers from all over the United States. Instead, she hired inexperienced teachers from the agencies with which she had close associations. These agencies received huge fees for supplying teachers to the district.

People who doubt that "reform" in DC was about money need to ask themselves why Rhee did this.

Posted by: Linda/RetiredTeacher | October 25, 2010 8:47 PM | Report abuse

Linda,

What is the proof? Could you get the FBI to investigate? Why not try? You don't, and won't, because you know there is nothing there. Just another smear and anti-change rhetoric.

Posted by: axolotl | October 26, 2010 10:29 AM | Report abuse

Axolotl:

No, I don't know if any laws were broken. I only know from what I've read that some fully qualified teachers with satisfactory ratings were dismissed and inexperienced teachers from Teach for America and the New Teacher's Project were hired in their places. These agencies received fees for these placements. Is this true? Did Rhee or anyone related to her profit from these fees?

To me this sounds mighty fishy. Don't you agree? The information is out there so if federal agencies want to investigate, they are free to do so. The only information I have comes from the Washington Post.

But let's move on to positive developments. I just viewed Jay's interview with Vincent Gray. Mr. Gray wants to provide infant and toddler education, along with universal preschool. He also wants to get the best teachers possible and support them in the work that they do. Now there's something that will help. I think I'll write him a check right now.

You are obviously a well-educated person who cares about education for all children. Surely you know that it takes a collaborative effort among students, parents, teachers and community to provide a high-quality education for everyone. Why don't you join us in demanding the best for DC students and children everywhere. Of course you are correct in saying that some teachers are ineffective but common sense should tell you that bashing them isn't going to help.

Posted by: Linda/RetiredTeacher | October 26, 2010 12:16 PM | Report abuse

Charters and TFA have been around for 20 years now, with lots of research to show they are not any more effective (and often less) that what's already out there. Nonetheless they are touted as successful mechanisms of reform. Newer reforms, such as merit pay, VAM, NCLB have been shown to be unreliable or even harmful, yet they continue.

Posted by: efavorite | October 26, 2010 1:13 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 

© 2010 The Washington Post Company