Network News

X My Profile
View More Activity


Posted at 9:30 AM ET, 12/12/2010

What international test scores really tell us: Lessons buried in PISA report

By Valerie Strauss

This was written by William J. Mathis of Goshen, Vermont. He is the managing director of the National Education Policy Center and a former Vermont superintendent. The views expressed are his own.

By William J. Mathis
For the 27th, government officials have yet again been surprised, shocked and dismayed over the latest international test score rankings. Secretary of Education Arne Duncan said, “We have to see this as a very serious wake-up call.” Former Reagan education official Chester E. Finn Jr. reported that he was “kind of stunned” by the results of the Program for International Student Achievement (PISA) results. In hyperbolic overdrive, he compared the results to Pearl Harbor and Sputnik.

The PISA tests were given to 15-year-old students by the Organization for Economic Cooperation and Development (OECD) in 65 nations and educational systems. Nine had higher average scores in reading, 17 in math, and 12 in science.

While ranking nations on test scores is a pretty sorry way to evaluate education systems, there is simply no reason to expect the results to have been any better than they were the last time we heard from this same chorus of surprised, shocked and dismayed pundits and politicians.

The reason is simple. Federal and state policymakers continue to embrace reforms that have little positive effect (if not downright negative effects) while ignoring reforms that make a difference. Buried within the PISA report is an analysis of educational systems that registered high test scores. Here are some of the less-reported findings:·

*The best performing school systems manage to provide high-quality education to all children.

· *Students from low socio-economic backgrounds score a year behind their more affluent classmates. However, poorer students who are integrated with their more affluent classmates score strikingly higher. The difference is worth more than a year’s education.

· *In schools where students are required to repeat grades (such as with promotion requirements), the test scores are lower and the achievement gap is larger.

· *Tracking students (“ability grouping”) results in the gap becoming wider. The earlier the practice begins, the greater the gap. Poor children are more frequently shunted into the lower tracks.

· *Systems that transfer weak or disruptive students score lower on tests and on equity. One-third of the differences in national performance can be ascribed to this one factor.

· *Schools that have autonomy over curriculum, finances and assessment score higher.

· *Schools that compete for students (vouchers, charters, etc.) show no achievement score advantage.

· *Private schools do no better once family wealth factors are considered.

· *Students that attended pre-school score higher, even after more than 10 years.

As OECD Paris-based official Michael Davidson said in National Public Radio comments, “One of the striking things is the impact of social background on (U.S.) success.”

Twenty percent of U.S. performance was attributed to social background, which is far higher than in other nations. Davidson went on to point out that the United States just does not distribute financial resources or quality teachers equally. In a related finding, students from single-parent homes score much lower in the United States than they do in other countries. The 23-point difference is almost a year’s lack of growth.

Our Educational Policies
Unfortunately, federal and state policies do little to adopt these factors that other nations have found so successful. Countless finance studies show that funding across our schools is inequitable and inadequate. Federal and state governments vaguely note this concern but actions do not match the rhetoric. Our treatment of economically deprived students is to house them in segregated schools and shunt them into tracked programs.

A number of “get tough” social promotion policies have been adopted in states even though we know they are harmful. Despite a clear research consensus, early education is still politically disputed. Tracking students still remains the national norm even as we know it increases the achievement gap.

As the federal government (under both Republican and Democratic administrations) has become even more top-down and prescriptive, local schools become less autonomous and less like our successful international counterparts. Finally, the push for privatizing public education through charters, tuition tax credits, vouchers and the like does not result in better test scores and has the effect of increasing segregation, and the inequalities that lead to low test scores.

The American Dream
The American dream is that all children have an opportunity to be successful no matter how humble their roots. Thus, the most troubling finding in the PISA results is the lack of “resilience” among our children.

OECD measured resilience by looking at the scores of the least wealthy 25% of students and seeing what proportion of these students have academic scores in the top 25% of countries with similar socio-economic levels. In the highest scoring nations, 70 percent of the students are rated resilient.

The U. S. figure is less than 30%. In a nation which sees the top 1% controlling more than 50% of the nation’s wealth and the collapse of middle class jobs, we face the specter of building a country of social, economic and educational apartheid.

Secretary Duncan calls the PISA scores a serious wake-up call for our economy and “international competitiveness.” But that is merely to misunderstand economics and global competitiveness. Due to our pursuit of ineffective and ill-focused educational and economic reforms , the rude disturbance of our slumbers is the slamming of the door on the American dream.

-0-

Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!

By Valerie Strauss  | December 12, 2010; 9:30 AM ET
Categories:  Guest Bloggers, Research, Standardized Tests  | Tags:  PISA, PISA scores, U.S. PISA scores, arne duncan, education policy, how u.s. fares in international rankings, international rankings, international test scores, obama education policy, pisa, the american dream  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Colleges start revealing early admissions decisions
Next: A Felliniesque education story

Comments

question:

Does anyone know if these tests were given to all "levels" of students in the international comparisons? For example, Germany gives a test to students when they are ten years old which determines whether they follow an "academic", "commercial," or "trade" track in school. Their educational future is determined at that point and it is very difficult to change tracks. I assume this is also done in other European countries. I have not seen any details about how and to whom the test is given in the European and Asian countries. It could make a big difference in the scores.

Are we comparing apples and oranges?

Posted by: mmkm | December 12, 2010 10:07 AM | Report abuse

mmkm, you have the right track. I checked on some in Northern Europe. Schools in some places have children attending 9 years of schooling that is mandatory. After that, you are right, they go different directions. Italy follows nearly the same route.

So this begs the question, are we comparing ALL our students to their "selected for college" students? Very good question.

Here's another position, at least mine anyway. Having lived in Italy and Greece, and travelled through many countries in Europe, our government and education system has it all wrong.

Stop poking our kids in the eye with testing that only serves someone else, and start evaluating our country and our living standards as part of the overall picture.

Posted by: jbeeler | December 12, 2010 11:16 AM | Report abuse

On tracking:

As a high school teacher I would have little or no desire to teach a class that wasn't tracked. I can't speak to lower grades, but a multi-level classroom would be a nightmare on a HS level. What I've found in ability tracks on the lower end of things is that there is not a culture of achievement. I am not sure if that comes from teacher expectations, student home lives, or some other area. That needs to analyzed before one totally tries to get rid of tracking.

Posted by: Mostel26 | December 12, 2010 11:35 AM | Report abuse

This "wake-up call" is done once every three years, and after we turn the alarm off, we go back to sleep.

The inequality is ingrained in our system of letting counties and cities control how they educate children in their districts. The wealthy bulk at the idea that their property tax money may be used to educate kids in other counties. They are okay the poor send their children to die for the country, but they are not okay if they have to spend their money to educate the children of the poor.

If we covered our education as often as we covered the football, basketball, or baseball, we would have a number one education system in the world.

How do we encourage kids to study math and science when we constantly remind them football, basketball, and baseball are where the money is.

Posted by: wu78754 | December 12, 2010 1:19 PM | Report abuse

Two questions for Mr. Mathis:

Most of your article seems to make sense, but the two following issues - and one that is not mentioned - would be worthy of greater explanation:

1) "*Tracking students (“ability grouping”) results in the gap becoming wider. The earlier the practice begins, the greater the gap. Poor children are more frequently shunted into the lower tracks."
It seems there is all kinds of tracking;
as several commentators mentioned above,
Germany has a very stringent system of
tracking when students get older, and yet
their students tend to do well.
Levels of most subjects need to be broken
down, and automatic tracking happens in
sequential classes like language and
math. I remember my 2nd grade reading
teacher splitting our class up into 3
distinct reading groups because we were
at such different levels.......?

I am thinking that we need some in-depth articles and discussion on what we mean by tracking in terms of these huge, comparative tests.

2. " *Systems that transfer weak or disruptive students score lower on tests and on equity. One-third of the differences in national performance can be ascribed to this one factor."
One third is an awful big percentage; as in issue #1, this merits a lot more attention and explanation. Why do the systems score lower? Are they generally more punitive to students and discouraging?
Or are they schools that just have such over-whelming socio-economic factors that
education cannot compensate enough?


Not mentioned or not looked at or not interpreted:
3.) What about the presence or lack thereof of a well-rounded curriculum? Does critical thinking come in here anywhere?
What about the co-relation of *musical, mathematical and linguistic abilities?

*There is some Education 'folklore' that many doctors are very good musicians - and that musicality aides an ear for linguistics. Or that the hand-eye coordination that develops when playing an instrument can translate to other tasks such as surgery, sports, and the visual arts.

Posted by: PLMichaelsArtist-at-Large | December 12, 2010 1:25 PM | Report abuse

Two questions for Mr. Mathis:

Most of your article seems to make sense, but the two following issues - and one that is not mentioned - would be worthy of greater explanation:

1) "*Tracking students (“ability grouping”) results in the gap becoming wider. The earlier the practice begins, the greater the gap. Poor children are more frequently shunted into the lower tracks."
It seems there is all kinds of tracking;
as several commentators mentioned above,
Germany has a very stringent system of
tracking when students get older, and yet
their students tend to do well.
Levels of most subjects need to be broken
down, and automatic tracking happens in
sequential classes like language and
math. I remember my 2nd grade reading
teacher splitting our class up into 3
distinct reading groups because we were
at such different levels.......?

I am thinking that we need some in-depth articles and discussion on what we mean by tracking in terms of these huge, comparative tests.

2. " *Systems that transfer weak or disruptive students score lower on tests and on equity. One-third of the differences in national performance can be ascribed to this one factor."
One third is an awful big percentage; as in issue #1, this merits a lot more attention and explanation. Why do the systems score lower? Are they generally more punitive to students and discouraging?
Or are they schools that just have such over-whelming socio-economic factors that
education cannot compensate enough?


Not mentioned or not looked at or not interpreted:
3.) What about the presence or lack thereof of a well-rounded curriculum? Does critical thinking come in here anywhere?
What about the co-relation of *musical, mathematical and linguistic abilities?

*There is some Education 'folklore' that many doctors are very good musicians - and that musicality aides an ear for linguistics.

Posted by: PLMichaelsArtist-at-Large | December 12, 2010 1:28 PM | Report abuse

VERY sorry! I asked 3 questions and said in the beginning that I had 2 questions for Mr. Mathis; then hit the post button twice.......
think I am pretty tired...

Is fatigue factored in when students take tests?

Posted by: PLMichaelsArtist-at-Large | December 12, 2010 1:30 PM | Report abuse

Great article.

By the way, Newsweek has a new article on Michelle Rhee and they are blocking all comments.

http://education.newsweek.com/blogs/crib-sheet/2010/12/06/why-michelle-rhee-isn-t-finished-with-school-reform.html

Posted by: educationlover54 | December 12, 2010 3:00 PM | Report abuse

Three issues - Who is being measured? In some countries is it may only be the students in the academic track. What is being measured? Do we believe it is a good idea to emulate the Korean and Japanese practice of double schooling college bound students with day schools committed to uniform progress, and evening schools, referred to as "cram schools," that cater strictly to preparing students to take their college admission test?
John Dickert
Mount Vernon Farms

Posted by: 12191946 | December 12, 2010 3:14 PM | Report abuse

Mr. Dickert summed up my initial question about who is being tested in the international community. My intent was not to raise the issue of tracking (although that is a valid issue to discuss), but to question who is being tested. Again, are we comparing apples and oranges?

Are the Europeans and Asians only testing the academic track?

By European standards, we do not currently track students (except for those who are severely learning disabled and a very few others). Many students who are not taking AP or IB classes do intend to go to college. And, remember, the tracking starts very early in other countries.

Posted by: mmkm | December 12, 2010 5:17 PM | Report abuse

Mostel26 worries me. His love of tracking being a proven failure. In fact, at every grade level the more differing students in the classroom, the greater the achievement levels - assuming competent teaching. Age-based grades are a huge problem in education since that system blocks student mentoring and the "teaching" component of learning. Tracking shuts off those educational paths completely.

The denial of American tracking (see mmkm's comment) is odd. Though lacking the formal structures of European systems American students are actually far more aggressively tracked. Those who fall behind in Kindergarten live with those labels throughout their K-12 experience, and are treated as second or third class citizens continually.

Posted by: irasocol | December 12, 2010 8:49 PM | Report abuse

I went to the OECD PISA presentation at the Dirksen Building earlier this week and was very impressed with the presentation. As I listened to the presentation, I was particularly impressed with the comprehensive analysis of the data. The presentation went far beyond simply comparing scores between participating countries and ranking them. The outcomes fit into four major areas for me.

1. The first view was the typical competitive comparison of the US students to the rest of the world. We are not at the top, and that is clearly a problem economically.
2. The second was the information about the results showing a significant population of the US testing below baseline level 2 “Level 2 is considered a baseline level of proficiency, at which students begin to demonstrate the reading skills that will enable them to participate effectively and productively in life”. Andreas Schleicher from OECD, who gave the presentation, pointed out that moving those low achieving US students above the baseline level would mean trillions of dollars improvement in the USA GDP.
3. The third view was looking at the size of the range from the lowest to highest scores within the US. (The narrower the range from lowest to highest score, the more educational equity in the country.) This range from lowest to highest was relatively high in the US. Countries that were ahead of us on the raw scores also tended to have a lower disparity (or more equity) across their populations. Related to this was my fourth observation.
4. The fourth was the priority of spending. In nearly all of the other “developed” countries, they spend more money attempting to raise the performance of low performing schools than they do on those schools that are already doing well. The US was practically the only country where the reverse is true.

As he was doing his presentation, Andreas also described which particular educational strategies appeared to have the most/best impact on improving student performance. It was reassuring to observe that every one of the practices the data showed contributed to improved student performance was a part of our Discovery and Innovation model.

Posted by: cheriallan | December 13, 2010 8:24 AM | Report abuse

Some responses to your questions on PISA scores:
1) Sampling - Several of you have asked if we are comparing ourselves to countries who only test their best scoring students. This is certainly true with number one scorer, Shanghai. Further, some tested countries do not have universal education through age 15 and others have significant drop-out and absenteeism rates. Sampling procedures within countries are not always transparent. On top of the sampling problems, there are a host of translation issues and ordinal scale problems. All in all, we shouldn't take national rankings too seriously -- that's not to say the data is totally worthless.
2) Tracking - I report what PISA says and a close examination by the reader is invited. There is a great reservoir of legitimate studies on tracking in the US that confirm what PISA found. In general, tracking does increase the achievement gap and fosters inequalities. Is it something we have welded into our culture? Probably. But we will not have equality if we teach to such different levels and continue to provide our subpopulations a lower level of education. Without changes, you can expect to hear our politicians "wake-up" again in three years with the next round of rankings.
3) Curriculum equality - There are huge differences between nations and within nations. NCLB research shows curriculum has narrowed particularly for low scoring schools in the US.
4) International competitiveness and education - This link is far more attenuated than what the political interpretations would suggest. For example, sub-prime mortgages, the exportation of manufacturing jobs, and the economic collapse were not caused by a poor education system. But that's for another column.

Bill Mathis, NEPC

Posted by: wmathis | December 13, 2010 10:53 AM | Report abuse

Mr. Mathis, regarding tracking, you state that tracking increases the achievement gap, but in which direction? Are you suggesting that tracking only depresses the bottom 25% of scores? Is it also possible that tracking increases the top 25% of score? I don't pretend to be a statistical expert for these data, but how does distinguish these effects within a single sample (i.e. the U.S.)?

On a more philosophical point, would you really WANT a culture of mediocrity? Without tracking, you're also dumbing down your smartest students -- they can't go at a pace that suits their abilities. I mean, were you ever a brilliant student stuck in a non-tracked class? It makes you want to bang your head against the wall -- that's how utterly dreadful it is. Frankly, I don't want my children stuck in non-tracked classes, bored to tears and not being challenged.

If we really want to elevate our economic success, we should focus on ALL students, instead of just worrying about the bottom students. The top students should not be ignored or "shunted" (to use others' terms) into classes full of less academically capable students. Let's face it: We are NOT all created equal -- some people are just brighter and more capable of succeeding in school than others, aside from any socioeconomic factors.

Posted by: rlalumiere | December 13, 2010 1:07 PM | Report abuse

It seems Korea and Finland scored very well. Wouldn't it be advisable for US policy makers to determine the common reasons for the success of these countries and to then use these features as design elements for school reform in the US? A cursory review indicates both have:
-a commitment to the success of every student
-universal early education
-extensive broadband access
-Education as a national priority

Posted by: danfrench | December 13, 2010 1:52 PM | Report abuse

You're right -- we think and act dumb in not looking to and then using research but instead thinking ideologically and acting on that. We need to think smart in generating a science base about what works and does't work, and then implement those findings.

There's more research findings that should be used.

Adolescents get sleepy later and need more sleep. So school start times should be moved back accordingly. When this has been partially done, attentiveness increased (especially in first period classes), and most important, grade and test scores meaningfully increased.

NIH research identified the most effective way to teach the core skills of reading. Use this.

NIH research has also provided the basis for a transformative intervention for children of disadvantaged children. The evaluations of educational interventions typically show small or no effects, and the small effects usually evaporate. One exception: Intervening with 20 hrs/wk of home visiting with 8 mos. to 3 years of age children of disadvantaged families to verbally interact with the little ones. This develops vocabulary, and vocabulary correlates better than .9 with verbal IQ. Results: At the 4th grade, average IQs of 100 and educational achievement at grade level. That's transformative for the children, and with enormous benefits for the county in international competition for ideas and jobs of the future, and for reducing delinquency and criminality, teen parenting (usually a disaster for the children), and other social ills.

As the transformation in the good that physicians can do when they have the results of science to use, education that uses research can also enable better living.

Medical research continues, and so should educational research. A National Institute of Education (NIE) with a size commensurate with the need (enormous) and whose science has the rigor of NIH research is necessary to enable further progress. Advocate for such a NIE.

Posted by: jimb | December 13, 2010 5:00 PM | Report abuse

The points made in your article lack relevance because you fail to quantify the data in analytical fashion; noting means and SDs would be nice; and you fail to provide the context so necessary in a diverse nation. You write, *Private schools do no better once family wealth factors are considered." Plainly, your statement is designed to promote the notion that high end private academies are equivalent to public schools. This is patently false. Scores are likely only similar, and to what degree you do not say, because of the selection effects by wealthy parents. The use of private schools is basically a vehicle to ensure that your child is attending school with the right type of peers. If you live in McLean, public schools are fine; if you live in Detroit, then only private schools will do. Since fewer wealthy kids will attend crummy public schools, the comparison of private to public along income streams is meaningless because you are simply comparing great public schools to great private schools.

As for tracking; Germany has a rich history of tracking and their economy is roaring by comparison to ours. So what is the problem with tracking? That some people don't benefit? Since when has the world been fair? If tracking is such a big and unfair deal, then why is it allowed in athletics? Why don't all kids earn a spot on varsity? Surely, athletics are less important than math and science?

Posted by: zylonet | December 13, 2010 9:56 PM | Report abuse

Forget Shanghai. Asian students spend eons memorizing and regurgitating--read posts on the NYT article of Westerners who have taught there. Yes, it is important to learn content, but if you can't apply it in a new situation, you're missing something big. The OFFICIAL POLICY of the Chinese government is being altered to encourage students learn to be more creative (read the writings of Dr. Zhao). They want to be MORE LIKE US (now that's ironic). The problem is their competetive nature doesn't let go of the test-prep mania so easily. So we have embraced the high-stakes testing insanity, and they have realized it is a mistake.

The country we should be looking at is FINLAND (ummm, #2 on the list--not too shabby). The students there don't start official academic schooling until they TURN 7; almost every child attends a high-quality (NON-ACADEMIC) pre-school where they are taught socialization and so on (think Vygotsky). Their school year is similar in length to ours; no Saturday school; no standardized tests. Their teachers are paid well; they differentiate instruction for all students. Their students are taught to THINK and to problem-solve. Students get lots of worthwhile projects. Students have universal healthcare and schools have lots of resources.

We (and other countries like G.B.) keep sending "delegations" over to Finland to see what they are doing that is so great. But apparently we don't really want to hear the answer (because it's not some quick fix that would make some US ed. company lots of money??) because we never seem to actually follow their example. Too bad. They seem to have a quality system where students excel and where they also all (or nearly all) actually like to learn (unlike many of the Asians--read the writings of many of the students themselves who are pressured to death by their families).

Posted by: MathEdReseacher | December 14, 2010 6:19 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company