Should we believe college rankings?

When you were in school, did you keep track of the number of times you raised your hand in class? Or, alternatively, could you recall how often you were engaged in “synthesizing and organizing ideas, information or experience into new, more complex interpretations and relationships?”

These are some of the questions on the just released 2009 National Survey of Student Engagement. The survey, as my colleague Dan DeVise reports, began 10 years ago, designed by Indiana University researchers to counter U.S. News & World Report's annual college rankings.

The idea is not to rank overall quality, which U.S. News purports to do, but instead tries to quantify what students are getting out of their college experience and how they spend their time. Results are used by schools to change policies and practices, not to mention by prospective college students checking out colleges.

For the 2009 survey, about 360,000 students at 617 U.S. colleges and universities answered the questions. Many of the nation’s leading public and private schools participate, but some of them don’t, including none of the Ivy League schools, nor the University of Virginia (it used to but dropped out), nor Johns Hopkins University, nor Northwestern University, to name a few. The University of Maryland system schools do participate.

The survey’s goal is commendable, no doubt. Any regular reader of this blog knows that I am not fond of rankings, especially the U.S. News college lists, which uses a ranking scheme that gives the largest value to how top college officials view the reputation of their competitors. (That’s data-driven for you.)
I wish I could have more confidence in the answers of the student engagement survey, affectionately called Nessie. But the problems that I see as a non-expert in polling seem significant enough to make me wonder about the results.

In this poll the average institutional response rate was 36 percent. Is that percentage representative of the school, or is there a particular type of student who would take the time to answer such a survey?

Oh yes, college students answered the questions. In what state, nobody knows.

Then there’s the issue of the questions and available response possibilities. Before I tell you what I think, it’s only fair to note that the survey’s Web site cites a 2004 study that says:

“Generally, students found the questions to be clearly worded and easy to understand. The number of items that prompted discussion [in student focus groups] was relatively small, less than 10% in most focus groups. (p. 240)

“The majority of students interpreted the questions in identical or nearly identical ways. (p. 247) ”

Excuse me, but interpreting the question in the same ways doesn’t necessarily mean that they could answer them with any great precision.

For example, let’s look at this question on the 2009 questionnaire.

During the current school year, how much has your coursework emphasized the following mental activities?

The answer options: Very much, Quite a bit, Some, Very little.

What do those mean? Quite a bit vs. some? Is 'very much' 2 or 4 or 6 1/2 mental activities away from 'quite a bit?' But I digress.

Some of the question prompts:

a) Memorizing facts, ideas or methods from your courses or readings so you can repeat them in pretty much the same form

c) Synthesizing and organizing ideas, information or experience into new, more complex interpretations and relationships.

e) Applying theories or concepts to practical problems or in new
situations


I'm sure if I were in college I would be counting the times I applied theory to practical problems. You too?

There are, of course, more concrete questions on the survey.

For example:

Mark the box that best represents the extent to which your examinations during the current school year have challenged you to do your best work

To respond, students are asked to mark a number from 1 to 7, with 1 marked “very little” and 7 marked “very much.”

Not specific enough for you?

There’s this:

Number of books read on your own (not assigned) for personal enjoyment or academic enrichment

Number of written papers or reports of 20 pages or more

Number of written papers or reports between 5 and 19 pages

The possible choices:

None
1-4
5-10
11-20
More than 20

I'm quite confident that all 360,000 participating students collected their papers and counted the pages. Aren't you?

Go ahead. Tell me if I am being too critical. Or not.

-0-

Follow my blog all day, every day by bookmarking href="http://voices.washingtonpost.com/answer-sheet/">http://voices.washingtonpost.com/answer-sheet/

And for admissions advice, college news and links to campus papers,
please check out our new Higher Education page at href="http://washingtonpost.com/higher-ed">washingtonpost.com/higher-ed
Bookmark it!

By Valerie Strauss  |  February 16, 2010; 12:00 PM ET
Categories:  Accountability , Higher Education  | Tags: higher education, student engagement survey Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Previous: Fairfax doublespeak on school openings and MoCo’s gamble
Next: Lessons learned from snow days

Comments

I think you are being too critical. One of the beauties of NSSE is that the questions it contains have a long, well-studied research history. We know, from decades of prior research, not only that they mean the same things to students, but also WHAT they mean. And for most of these questions, we know that the answers are strongly related to student learning.

And remember nothing, absolutely nothing, in the U.S. News rankings is even dimly related to student learning. Smushing together a whole lot of factors that have nothing to do with student learning into a single ranking doesn't get us any closer to whether students are learning. NSSE does.

And better you should ask why colleges and universities are not using NSSE. Why aren't they interested in using the best instrument we have for broad assessment of student learning? Are they wary of testing their prestige against real evidence?

Douglas Bennett, President, Earlham College

Posted by: dougb5 | February 17, 2010 11:11 AM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company