Network News

X My Profile
View More Activity


Posted at 11:25 AM ET, 10/ 1/2009

Berliner: Why Rising Test Scores May Not Mean Increased Learning

By Valerie Strauss

The Answer Sheet asked prominent researcher and educational psychologist David Berliner of Arizona State University to explain why rising standardized test scores may not mean that students have learned more. Here is his post:

By David Berliner

A rise in test scores leads most people to believe good things are happening in their schools. Not unreasonably, politicians and parents alike infer that students have learned more when test scores go up. But since the No Child Left Behind (NCLB) law was passed that inference may be unwarranted. Sadly, there are numerous reasons why rising test scores may not be related to increases in student learning.

1). Virtually all states have changed the passing score on tests so that more children are classified proficient.

NCLB is bizarre since it requires that 100 percent of the children at a school need to be “proficient” in reading and mathematics by 2014. But at what score is one proficient?

The choice of a score for “passing” a test, or for being labeled “proficient,” “basic” or “exemplary,” are value-laden choices. They have little to do with statistics and measurement, and are, instead, political decisions.

Think of it this way: If your students, teachers, and schools were faced with inevitably and inescapably being labeled as failures under a law that is impossible to meet, but could instead be seen as successful by dropping the score at which students are classified as proficient, wouldn’t you be tempted to do that? Almost every state has done exactly that.

By fiddling with passing scores more children are declared proficient. But the increasing percentage of students declared proficient does not necessarily mean that students’ scores have actually risen, or that greater learning has taken place. It may only mean that more students are classified as proficient because the score needed to be classified “proficient” was lowered.

2). School districts across the nation engage in excessive, perhaps unethical, and, in some cases, illegal test preparation. This results in higher test scores, but not necessarily greater learning.

NCLB has resulted in large increases in test preparation activities to ensure that student scores go up, as required by the law. But the test was built to assess learning under “normal” conditions, not conditions in which students were drilled daily in tasks known to be similar to those on their state test. Normal conditions do not mean daily or weekly testing with exams suspiciously similar to those used by the state.

Data from all over the country suggests that it is not uncommon for 20-60 school days per year to be spent in test-preparation. Children can certainly be trained to answer questions in a certain way if they are drilled enough on items like those that will appear on their test. In that way, test scores can be made to increase. But that is not due to education, it is due to training. Under such circumstances it is not clear that authentic learning has taken place.

3) Familiarity with the objectives and the items on a test invariably results in increased test scores.

Teachers and administrators are not fools. If a test is given every year with the same objectives, built to the same curriculum standards, and using many of the same items from one administration to the next, teachers and administrators come to know what will be on the test.

Unless the testing company employed by the state is willing to change items frequently, test scores will inevitably rise every year. Test score increases may be due to teachers’ knowledge about what is on the test, or because students really learned more than the year before, but we do not know which explanation is appropriate.

It is expensive to change test designs and items frequently, so that usually is not done. The inevitable result is that a larger and larger percent of the children in a state score above the average that was obtained the first time the test was given.

The result is called the “Lake Wobegon Effect,” named after Garrison Keillor’s mythological Lake Wobegon, where “all the woman are strong, all the men are good looking, and all the children are above average.”

4) The test items we use do not tap the knowledge we really want to assess.

Most NCLB tests consist of many multiple-choice items, rather than essays or other techniques for the assessment of complex performance.

The reason for that is simple: multiple-choice items are cheap to produce and cheap to score. Also, by including lots of such items during any one testing session a test becomes more reliable. That is, the scores obtained are more dependable as indicators of whatever is being measured. In contrast, because of the time they take to answer, you can only have one or a few essay items per test session.

Because of the small number of essay items on a test, essay exams are usually not reliable indicators of learning. Unreliable tests should never be used to make decisions about students, teachers, or schools.

So we don’t see much in the way of essay testing. Furthermore, essay test items are much more expensive and time consuming to score. It boils down to this: Extensive reliance on multiple-choice questions makes it harder to be sure that deeper, more complex learning has taken place.

5) Afraid they could be fired or their schools closed because of NCLB test scores, district and school administrators invent ways to prevent the poorest performing students from taking tests.

Students are dropped or pushed out of school, certain students are suspended, and some students are moved to other schools mid-year so their scores will not be counted.

For example, because they were expected not to do well, more than 500 students from high schools in Birmingham, Ala., were dropped just before state testing. And in New York City political leaders had to apologize for policies that pushed out thousands of academically weak students from the schools. Pushing out the weakest students raises district or school scores.

Meanwhile, high performing students are pushed to take tests that they previously passed a second and a third time so that the average scores of a school or district will go up. Manipulation of test scores makes interpretation of those scores problematic. Untrustworthy data means we do not know if higher scores really equal greater learning.

Furthermore, the pressure that NCLB puts on teachers transforms the relationship between teachers and students from a caring one into an instrumental one. With high-stakes testing children are too frequently seen positively only if they can increase school test scores, and they are too often seen negatively if they cannot. A child’s worth has become their test score.

6) It is common for scores to go up because of cheating. For example, there are companies that look for anomalies in test scoring. They often find incidents such as a low-scoring student suddenly getting seven items right in a row, or a class in a low-performing school suddenly outperforming classes in a neighboring high-performing school. These may or may not be instances of cheating, but several hundred of these anomalies are associated with NCLB tests in many states.

In Texas, the State Department of Education refused to investigate one such discovery. This was not a surprise since it was uncovered that the head of the State Department of Education had for many years been turning in false scores from the district in which she had previously been a superintendent.

Wesley Elementary School in Houston provides another example. It won accolades for teaching low-income students how to read. The school was even featured on an Oprah Winfrey show about schools that “defy the odds.”

In 2003, Wesley’s fifth graders performed in the top 10 percent in the state on reading exams. The next year, as sixth-graders at the Williams Middle School, the same students fell to the bottom ten percent in the state. Confronted with data the Wesley teachers admitted cheating was standard operating procedure.

Conclusion: It is likely that there will always be some corruption associated with the people and the tests used to assess learning when so much pressure is on administrators and teachers to increase test scores.

This means that when scores do go up, we need to be wary. We need to investigate whether the rise in test scores is a real indicator of greater learning or some form of deception.

Unfortunately research suggests that deception and cheating in contemporary American culture, including our schools, has become more acceptable. Among the many problems associated with this cultural shift is that it is not easy to interpret the test scores associated with NCLB. We are no longer certain that a rise in test scores means that more student learning took place.

Caveat Interpretis! Let the interpreter of scores beware!

-0-


David C. Berliner is co-author with Sharon L. Nichols of Collateral Damage: How high-stakes testing corrupts America’s schools (2007). He is also co-author with Bruce J. Biddle of The Manufactured Crisis: Myth, fraud, and the Attack on America’s Public Schools (1995).

For more on cheating in contemporary American schools see Callahan, D. (2004): The cheating culture: Why more Americans are doing wrong to get ahead; and Nichols, S. N. and Berliner, D. C. ( 2006). The Pressure to Cheat in a High-Stakes Testing Environment, In E. M. Anderman & T. B. Murdock (eds): The Psychology of Academic Cheating.

By Valerie Strauss  | October 1, 2009; 11:25 AM ET
Categories:  David Berliner, Guest Bloggers, Learning, Standardized Tests  | Tags:  David Berliner, No Child Left Behind, Standardized Tests, high-stakes standardized tests, school reform  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: THE GROUP: The Extracurricular Activity Problem
Next: Moms: Which Years Would You Like to Stay Home?

Comments

I agree with David Berliner regarding standardized testing. I have observed another mess with my own children during their high school years. Since they changed schools during high school yet found the same problem, I believe this is not an isolated issue. My sons visited libraries often to choose books of their special interests, typically, of a scientific nature. About half a dozen times when given research assigments, they were allowed to choose their own topics related to the general subject. They certainly jumped on the opportunity to further their knowledge about a topic and enjoyed delving into the projects with enthusiasm. However, even thought they were in the "advanced" classes, their classmates typically chose subject matter of a much simpler nature. The students were usually given about 10-15 min. later on to give a presentation to the class. My sons were given poorer marks than the students who gave junior high level info because this was easier for the teacher to follow (probably). No, I am not mistaken, this was the trend. Even tho' they gave clear presentations, explained the info well and altered portions to be sure that a layman could understand (if paying attention), no go for the grade. Though they continued to pick more difficult topics for their research projects, even knowing their grades would take a hit, they enjoyed learning. They were sometimes confronted with the same scenario in undergrad classes. They have strong intrinsic motivation and love of learning. Now both in grad school (in the sciences), their dedication to excellence is finally appreciated - by their professors and fellow grad students. But, really, it is a sad reflection on the state of learning in America.

By the way, neither did so great in the arena of standardized testing, but wow, can they analyze!

Posted by: shadwell1 | October 1, 2009 2:08 PM | Report abuse

I still remember the first time I realized that schools were cheating. It occurred about ten years ago when I was home with the flu. While channel surfing, I was surprised to find that I could watch successful schools being honored by state legislators. I was curious to know about these schools so I tuned in.

One school was honored for taking its children from the 12th percentile to the 90th (or something like that) but when the state senator asked the principal how her school managed such a huge improvement, she couldn't give one cogent reason. She just said things like "Our teachers really worked hard " or "We stress good behavior."

Another time a school district won the prestigious Broad award. I congratulated a teacher and asked, "How did you do it?" She responded in this way: "We placed desks around the desk of a high achiever and then pushed them together." Was that a joke? I really don't know.

After New York City evaluated most of its schools as "outstanding" I'm hoping that journalists are catching on at last. A huge fraud is being perpetrated on the American people and it can cause a lot of damage if it continues much longer.

Posted by: Linda/RetiredTeacher | October 1, 2009 7:11 PM | Report abuse

Test prep = lighting a match under the thermometer and claiming you have raised the temperature of the room.

Posted by: skrashen | October 1, 2009 9:14 PM | Report abuse

Thank you to the Washington Post for finally printing an article that examines the testing issue a little further and to David Berliner for clearly explaining the true details. I would like to add a comment about testing in math.

People tend to think that math is so straightforward--the kids either know the stuff or they don't. When they know the material, they get the right answer; when they don't know the material, they get the wrong answer--very cut and dry.

However, this is a MYTH. Just because a student gets the right answer, it doesn't mean s/he 'knows' the material. Teachers increasingly teach recipe math ("Just learn these steps, and you'll get the right answer"), so most students never really learn to think mathematically. After a time, even those learned procedures are forgotten because they were never learned with any kind of understanding.

Posted by: MathEdReseacher | October 5, 2009 9:43 PM | Report abuse

I just completed my dissertation titled "Teachers Cheating on Standardized Achievement Tests: The Perceived Causes and Effects." Cheating is definitely happening in Arizona and I venture to say it is probably happening in every other state in the country! The use of test score results to label, to reward, to punish teachers, students and schools is ridiculous and unfair. We have become a nation obsessed with testing and it is up to us, as educators, to stand up and do something about it.

Posted by: srideau | October 6, 2009 12:44 AM | Report abuse

An example of cheating on standardized tests that was uncovered in Montgomery County, Maryland - the cheating by adults was exposed by the children.

Newsweek - Bitter Lessons: The Kids Were The Heroes In The Scandal At Potomac Elementary. What Happens When Role Models Teach Dishonesty

By EVAN THOMAS AND PAT WINGERT | NEWSWEEK

The first hints of something wrong at Potomac Elementary came from the kids. Whispering to one another in the hallways and on the playground, then telling their parents after school, a few fifth graders began describing the peculiar behavior of their principal...

...But on a deeper level, the Potomac scandal is a morality play with a disturbing twist: the heroes were the children who had the courage to question the ethics of the very people who were supposed to be teaching them the values of honesty and integrity...


http://parentscoalitionmc.blogspot.com/2009/10/newsweek-on-mcps-what-happens-when-role.html

Posted by: jzsartucci | October 6, 2009 7:11 AM | Report abuse

The comments to this entry are closed.

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company