Network News

X My Profile
View More Activity

Study: highly-rated professors are. . . overrated

How does a university rate the quality of a professor? In K-12 education, you have standardized tests, and those scores have never been more widely used in evaluating the value added by a teacher.

But there's no equivalent at the college level. College administrators tend to rely on student evaluations. If students say a professor is doing a good job, perhaps that's enough.

Or maybe not. A new study reaches the opposite conclusion: professors who rate highly among students tend to teach students less. Professors who teach students more tend to get bad ratings from their students -- who, presumably, would just as soon get high grades for minimal effort.

The study finds that professor rank, experience and stature are far more predictive of how much their students will learn. But those professors generally get bad ratings from students, who are effectively punishing their professors for attempting to push them toward deeper learning.

The study is called "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors." It was written by Scott E. Carrell of the University of California, Davis and National Bureau of Economic Research; and James E. West of the U.S. Air Force Academy

It uses as a laboratory the Air Force Academy, where students are randomly assigned to courses such as Calculus, each taught using an identical syllabus. All students are required to take specific follow-up courses. So, the researchers were able to study how each professor fared in producing results for his or her students, and how the same students did the next semester, and so on.

The findings are, to say the least, counterintuitive. Professors rated highly by their students tended to yield better results for students in their own classes, but the same students did worse in subsequent classes. The implication: highly rated professors actually taught students less, on average, than less popular profs.

Meanwhile, professors with higher academic rank, teaching experience and educational experience -- what you might call "input measures" for performance -- showed the reverse trend. Their students tended to do worse in that professor's course, but better in subsequent courses. Presumably, they were learning more.

That conclusion invites another: students are, in essence, rewarding professors who award higher grades by giving them high ratings, and punishing professors who attempt to teach material in more depth by rating them poorly.

"Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions," the authors write, "this latter finding draws into question the value and accuracy of this practice."

Please follow College Inc. all day, every day at washingtonpost.com/college-inc.

And for all our college news, campus reports and admissions advice, please see our new Higher Education page at washingtonpost.com/higher-ed. Bookmark it!

By Daniel de Vise  |  June 11, 2010; 4:08 PM ET
Categories:  Administration , Pedagogy , Research , Students  | Tags: high-rated professors are worse teachers, popular professors teach less; professor ratings; professor evaluations; professor research study; professor experience tenure  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Former U-Md. student, porn actor dies in cliff plunge
Next: New CUA president: not a cleric, but a 'man of faith'

Comments

Interesting finding. It points to the peril of a single metric. Sounds familiar? Standardized testing as a single metric.

Posted by: knowledgenotebook | June 11, 2010 4:53 PM | Report abuse

A possible counter argument would be that the subsequent fall off in performance was due to the de-motivating effect of a less appreciated teacher, while a perceived better teacher in the subsequent course, from the students perspective, would have a motivating effect and better results.

Posted by: cbum1 | June 11, 2010 11:37 PM | Report abuse

Put enough of them at a University and what do you get? Yale, Harvard, and the rest of the overpriced ivies.

Posted by: Nymous | June 11, 2010 11:46 PM | Report abuse

@ cbum1

I understand the thinking from your comments, but think you may have misinterpreted the study and results. Fall off in performance only occured in subsequent classes taken by students who gave higher evaluations and got higher grades in the foundation course. Lower evaluations were given to professors who laid the foundational knowledge that benefitted students in subsequent courses. Positive evaluations were given to teachers who gave good grades, however the foundation they laid was not as strong because their high scoring/positive evaluating students didn't do as well as students who did poorly and rated poorly in the same subsequent courses. After glancing at the full paper it appears that the analysis they used allowed them to ascertain what predicted the outcomes. However, you could always replicate their study and see if you're right.:-)

Posted by: TheNuttyProfessor | June 12, 2010 12:23 AM | Report abuse

Big deal professors are usually a joke. Their classes are taught by students who are paid little while the "name" professors receive many times what they are worth. The professors have students write textbooks for them, have them published and then require the books for their classes.
If you want an education, go to a community college for two years. Save your parents a lot of money and get a good education.

Posted by: hurleyvision | June 12, 2010 12:43 AM | Report abuse

Why is this "counterintuitive"?

You can go on to any campus in the country and talk to students for 15 minutes and you'll know who the highest rated professors are. There's two things they all have in common.

1. A very entertaining lecture style
2. Plenty of A's and nobody who completes the assignments gets less than a B.

How much is learned never enters into it!

Posted by: corco02az | June 12, 2010 7:10 AM | Report abuse

Corco02az is exactly right. Entertainment and easy grades are the criteria of students in rating a teacher--not how much they learned.

In deciding whether to take a class students have a third criteria--the time of day and days of the week of a class--with something like late morning/early afternoon on Tuesday through Thursday the ideal.

Posted by: cmckeonjr | June 12, 2010 9:15 AM | Report abuse

I'm not sure some of the commentators know what they're talking about.

I've been a professor for 11 years now, teaching political science (which is very different from the calculus courses used in the study).

I've never had a "student teach the course for me." Most of us don't have graduate students teaching for us-- at least this is the case at a regional comprehensive state university (not a flagship).

Having never written a textbook (I focus on writing books/articles on my research interests), but knowing several who have, I can tell you that professors don't have their students write their textbooks. Writing a good textbook is not an easy undertaking.

Over the years, I've had courses where I rated relatively poorly and courses where I've rated very well. My student evaluations have improved markedly over time, but there are courses where I'm tougher (or do less innovative pedagogy) and the evaluations fall off. So, perhaps, there is something to the idea that entertainment value and ease in grading is important.

However, I can report that my colleagues and I have courses where students are challenged and the grades aren't distributed toward the positive end and these colleagues do very well on their student ratings.

So, while on average easy grading might lead to higher course evaluations, challenging students and using innovative pedagogy also yields higher evaluations. Certainly the former is the easier path for many professors to higher evaluations, but many of the experienced professors I work with take the second path.

I also agree that, perhaps, quantitative student evaluations should probably count less, overall, for promotion/merit/tenure. Peer evaluations are also an important metric, as are examples of student success and the qualitative statements by students.

I can tell you from experience, however, that really poor student evaluations are a marker of really poor teaching, which may include poor pedagogy, unclear expectations, and little learning. One way to forestall this outcome as a professor is to be really easy. Thankfully, most of us are professionals avoid taking that road.

At least those are my experiences as a professor and evaluator of other faculty (for tenure, etc.), with all apparent biases.

Posted by: approf_ohio | June 12, 2010 12:46 PM | Report abuse

I'm a college prof with lots of teaching experience. I suspect that this study is mixing up some different populations. There's the easy grader+entertainment issue mentioned, which is a sure-fire way to boost one's ratings.

However, I do NOT believe for a minute that faculty rated low are all teaching better material. Some are simply harder as noted, but others are not good instructors. One thing that WILL guarantee low ratings is to be disorganized as an instructor, or having an otherwise disorganized class, which may not be the direct instructor's fault, e.g., if there's a standard curriculum, required textbook, etc.

In general, I don't like ratings as I think they are one more thing that changes the relationship between teacher and student towards a customer service model, but occasionally they do provide useful feedback, but I'd rather ask for it during the course of the semester, though, when I can do something about it.

Posted by: JV17 | June 12, 2010 2:01 PM | Report abuse

This is a surprise to whom? The doofuses "educated" by those charming "Give-'Em-All-an-A" teachers? Has no one at the POST worked with the kids this system produces? The ones who can't spell, think or understand what they read? Or am I so old I don't grasp all the managers these days are as ignorant/clueless as their hires?

Even the supposedly "elite" students from the Ivies. But maybe they're the MOST ignorant--because they fit a demographic or Mommy and Daddy have made REALLY big contributions to get Baby admitted.

Young Americans are poorly educated? Duh.

Posted by: mftill | June 12, 2010 2:25 PM | Report abuse

This is a surprise to whom? The doofuses "educated" by those charming "Give-'Em-All-an-A" teachers? Has no one at the POST worked with the kids this system produces? The ones who can't spell, think or understand what they read? Or am I so old I don't grasp all the managers these days are as ignorant/clueless as their hires?

Even the supposedly "elite" students from the Ivies. But maybe they're the MOST ignorant--because they fit a demographic or Mommy and Daddy have made REALLY big contributions to get Baby admitted.

Young Americans are poorly educated? Duh.

Posted by: mftill | June 12, 2010 2:25 PM | Report abuse

I have been teaching at the graduate and undergraduate levels for 40 years. I tend to agree with the article's conclusions that if you teach the hard stuff and go into depth you get lower ratings. But I think there are (at least) two further variables. (1) Rapidity of feedback: students want you to quickly grade assignments and exams, and return them quickly. Delayed feedback is worthless and the students know it. (2) Outreach: If you teach the hard/deep stuff you need to reach out to students to assist the slower learners. Students know whether you care if they are getting the stuff and if you make the effort, they will think more of you.

The above two canons should not only improve evaluations by students but should also have better and more lasting educational outcomes.

I, too, have had good evaluations and bad ones. But when I follow the above two canons, I get better ones than when I don't.

It does not mean much to talk about evaluations in terms of "easy" and "hard" instructors. Evaluation must be considered in the light of the full range of instructional inputs. It should be no surprise that better ingredients make better pies and cakes.

Posted by: makrain | June 12, 2010 2:35 PM | Report abuse

Ratemyprofessors.com site - surprising how often "great professor" goes with "easy exams," "hard to fail class if you show up." Not always - as article and comments note, some profs are bad, unorganized. But easy exams often assure high marks.

Posted by: calvinconz | June 12, 2010 4:35 PM | Report abuse

I taught undergraduate and graduate business courses for 20 years. My experience is similar to, though not exactly the same as, the findings of the research article. The difference is in the student population.

I taught quantitative courses that sometimes used calculus. When I had students who were mostly serious, I received excellent evaluations. These were mostly of seniors who were ready to graduate and were looking for jobs or had jobs lined up. They were interested in being competent in their jobs.

When I had students who were mostly not serious, I received horrendous evaluations. These were mostly freshmen who were interested in having fun and blowing off classes. I have comments in ratemyprofessor.com that say it is too hard to get an A in my classes.

My peeve is with the administrators. It seems that due to financial pressures, they are accommodating the students much more and putting subtle pressure on faculty to grade inflate or otherwise appease the students. I don't know about the disciplines that include sociology, psychology, anthropology, English, art history, etc. I DO know that in the business (specifically accounting and finance), engineering, and computer science disciplines, faculty can earn 3 times their academic salaries in the private sector (though work hours will double). I do not at all understand how pressuring faculty to dumb down courses will benefit anyone. Students do worse in later courses and in graduate school, the school's reputation takes a negative hit in the eyes of employers, and the better faculty walk to the private sector. If there are any university administrators reading these comments, please clue me in because I really, REALLY, do not get it.

Posted by: Edwin2 | June 12, 2010 5:47 PM | Report abuse

I'm not sure I buy the interpretation of this study. Most students value professors who actually teach them something, even if those same professors are tough graders. But being a challenging teacher and a tough grader do not necessarily go hand-in-hand. The lowest grade I got in graduate school years ago was from the worst professor I had. She attempted to make up for her inadequacies by devising "hard" tests, which were actually just poorly constructed. Finally, student evaluations should have some, very limited relevance in the evaluation of a professor. Anyone who puts any value in the Rate-My-Teacher or Rate-My-Professor sites is not too swift.

Posted by: Jennifer88 | June 12, 2010 6:03 PM | Report abuse

Student evaluations may have their limitations. But studies like the one alluded to are generally close to worthless. An article about an unnamed study with no attempt to objectively evaluate that study's credibilty is no better than random gossip.

Posted by: dnjake | June 12, 2010 6:38 PM | Report abuse

My 25 year experience as a college professor partially supports some of the conclusions of this study. However, the use of the Air Force Academy as the "laboratory" may make the generalizability of the findings suspect. I have found that the degree to which students simply give high ratings to easier graders varies considerably from school to school. At a large state university where I taught, and served on a committee to review colleague teaching, I often found that instructors who gave higher grades also got higher student ratings. But at a small liberal arts college where I taught I saw no such correlation. At the small liberal arts college many student evaluations expressed appreciation for professors that "challenged" them and prompted them to learn more. So a big question for such a study is: "For what school, and what student population, does this apply?"

But the most important issue is not even what the student evaluations show, but how they are used by the educational institution. Schools that rely heavily on student evaluations to rate professor teaching are guilty of lazy, and shameful, bureaucratic short-cuts. Student evaluations can provide useful feedback for consideration. They should never be a substitute for peer review and other multiple measures of teaching and mentoring effectiveness.

Posted by: mythos1 | June 12, 2010 7:27 PM | Report abuse

Popular teachers are often not the most effective teachers at promoting student learning. That's been obvious to me observing numerous teachers, and it was obvious to me when I was a student.

Student evaluations of teachers typically end up being expressions of how the students feel about themselves, and especially of how they feel that they are doing in a course. One of my college professors actually put the student evaluations to the test. He taught two of the same course, and taught both in entirely the same way. He put the same kinds of comments on student papers, and the only difference was the grades assigned to the papers and exams. One class he graded harder, and the other much easier. In the class graded hard, a lot of students expected to get grades in the B and C range. In the class graded easier, people mostly expected A's, with a few B's tossed into the mix. The results of the evaluations? The class expecting mediocre grades rated the professor as a horrible teacher; the class expecting A's thought that he was the best teacher ever.

Students want to be coddled, and they want to feel good about themselves. Teachers who give high grades to average work will be heralded as great teachers in student evaluations. Teachers who grade work honestly will be panned, even if they are encouraging, positive people in the classroom and in feedback on student assignments.

Posted by: blert | June 12, 2010 8:23 PM | Report abuse

After teaching for over a decade in the US, I tend to agree with the author of the article without any reservation.

“Student evaluation" may have had played an important role in the American education in the past, now I believe, it has started hurting it.

Else why would American student fail in competing in the US, let alone globally!

But it is beyond one academic institution to stop the practice. I believe American education cannot be improved until one simple action is taken: declare the “student evaluation” practice ILLEGAL!

It has become a practice among the students and faculty “I scratch your back, you scratch mine”! Students must not be “customers” but should be the “knowledge seekers.”

Earlier the practice of “student evaluation” is scrapped the better it would be for the American education!

Posted by: samantas | June 12, 2010 8:54 PM | Report abuse

A problem with student evaluations is that they are usually filled out in the last few minutes of the class, and as soon as you're done, you can leave. So students rush through them, filling out the bubbles but not writing any actual thought-out additional comments that would be useful either for professor assessment or for the professor to improve themselves.
Tech is in the process of switching to online evaluations, which are good because they can be filled out at students' convenience, so they can actually put some thought into it, or just not do it, if they don't have anything substantial to say.

I don't think student evaluation should be entirely eliminated, because I know many professors value the comments (particularly the additional comments, rather than the filled-in bubbles). They've told us so. I think the bubble sheets are relatively useless, and get filled out fairly arbitrarily.


Posted by: sarahee | June 12, 2010 11:22 PM | Report abuse

Of course this study is correct in its findings.

Teaching a difficult task requires hard effort. Not everyone will get it. Those who do not will vent their spleen in professor evaluations.

Teaching garbage and making it entertaining while simultaneously telling students how brilliant they are and rewarding attendance with an A or B is a surefire method to get positive feedback.

Ultimately, we all need to decide whether we want to learn the hard lessons in life.

Posted by: tacheronb | June 13, 2010 12:05 AM | Report abuse

Seriously? This study is exactly the justification that "unpopular" profs need to continue not working on their teaching, hiding behind the "the students are too whiny and lazy" excuse. As a third-year faculty, I work my students into the ground, but continue to get excellent reviews. Part of the solution? Be fair, be explicit about expectations, and don't be afraid to rework teaching approaches if what you've been doing isn't cutting it or isn't communicating to the students.

The correlation between high marks and student performance could very well be the fault of a bifurcated educational philosophy, where students are taught to be specialists instead of comprehensive thinkers, or the fault of fluff teachers, or due to radically unfamilliar material. Take your pick. But to bag on "popular" professors, to my mind, is symptomatic of sour grapes, professors unwilling to work on their own teaching for whatever reason, hiding behind "difficult material" or their own over-inflated estimation of their communication abilities.

IMO.

Posted by: wacoan | June 13, 2010 10:01 AM | Report abuse

I think it's useful to read the entire study, which is posted here: http://homepages.uconn.edu/~leh06001/files/2312w/Carrel%20and%20West%202008.pdf.
The problem with the over-reliance on student evaluations for tenure and promotion decisions is that students often don't know the difference between being entertained and being educated--it's especially true with this generation of students. I feel successful when they come back to me years later unsolicited and thank me. That's when I know I've been successful, or at least have played a small part in the student's success.

The question is--how can an RTP committee look at student retention and application of course material years down the road? I think that's the real point of the article--how can we best measure the success of a class? Happy students are not always better educated, I agree. But I don't think I'm willing to conclude that unhappy students are--I think we need to have a truly multidimensional approach to evaluation. In the context of an increasingly corporate approach to higher education, I have less and less confidence that we'll come up with anything that's more than a pretext for doing away with the tenure system. I honestly believe that students do not benefit from an insecure, underpaid, and stifled faculty. Recall that seniority is correlated with increased student learning--but what so many university administrations are striving for is a revolving door of contingent labor. Frightened little mice are not going to have the courage to give students the grades that they truly earn, since if the "customers" complain, the instructor may not be retained. That's the real issue that this article, and others like it, missed.

Posted by: fcsprof | June 13, 2010 10:28 AM | Report abuse

There may be a confounding factor here. Sometimes the "popular professors" attract weak students.

I don't know if this study accounted for the intelligence of the students (as measure by, e.g., college entrance scores, grades in previous classes, etc.)

Posted by: blueollie | June 13, 2010 8:57 PM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company