Network News

X My Profile
View More Activity

Posted at 12:24 PM ET, 02/ 1/2010

Challenging Jay’s Challenge Index

By Valerie Strauss

Someone has to say it, so I will.

Today my esteemed colleague Jay Mathews published his Washington region version of the Challenge Index, the annual exercise in which he ranks high schools.

How does he do this? With this formula that he devised years ago: The number of Advanced Placement and/or International Baccalaureate and/or other college-level tests taken by all students at a school, divided by the number of graduating seniors.

That’s it. Is it enough? I don't think so.

Before I go further, let me say this about it: It is true that there are very few measures, if any, that allow you to compare schools with different budgets, demographics, teacher pay rates, etc. This, for all its flaws, is one.

And outside the Washington area -- amazingly, 77 percent of high schools around here rank in the top 6 percent of U.S. schools on Jay's national Challenge Index -- there are thousands of schools that put obstacles in the way of kids who might profit from more challenging high school courses. The Challenge Index challenges these schools to change their ways.

But still.

On his blog today, Jay says that the “outrage and acclaim” that swirls around his index -- and there is a lot of both -- “usually swirls around the issue of whether ranking schools is good for you.”

Actually, it swirls around more than that. It isn’t just about the value of ranking, which has become an obsession in this country, so much so that few of them have any meaning.

It is about his methodology.

Judging a school by a single measure is incomplete. And even though Jay says he isn’t judging the QUALITY of the school, but rather how it is challenging its students, readers view his rankings as a judgment of quality.

Besides, there are lots of ways a school can challenge students other than offering AP and IB courses. And some of the schools on the list have serious problems, such as high dropout rates and wide gaps between the achievement of whites and Asians and everybody else.

Jay says that his intention is to see how regular high schools challenge their students academically. That’s why he excludes magnet schools and schools that require admission criteria for more than half of their students (though he does include a school that requires all of its students to take AP classes, and that school turns out to be No. 3 on the new list).

And he says that he doesn’t think his index has had anywhere near as much influence on the practices of schools as some of those schools actually say.

Stop your kidding, Jay.

Everybody in the education world knows your index. Schools that wanted to move up on your list began offering more of these courses, sometimes to populations that may not have seemed ready. Because your index only considers the number of AP tests taken, and not the actual scores, schools put as many kids into the test pipeline as they could.

One graduate of Bethesda-Chevy Chase High School and parent of a BCC graduate said he thinks the methodology is suspect, but because the school always does well on the rankings, he loves it! His child was strongly encouraged by teachers to take extra AP and IB tests in what seemed to be a clear attempt to goose the numbers.

But still, I realized recently that maybe it isn't your fault. I realized that I should stop blaming YOU for all of this (though as I recall, you did not accept my forgiveness).

I was thinking about parents who push their kids into spending every waking hour trying to beef up their college application accomplishments so the Ivy League will welcome them. They should stop driving their kids insane.

And then I thought of your Challenge Index and realized that the administrators didn’t actually have to react to your list. They could just look at the ranking as another in a long list that either don’t tell us much, or tell us one piece of a complicated puzzle, and not fall over themselves to climb higher on your list.

Though I have to say that it must be fun doing this every year -- having people tell you they think your list is great or that they think it is the worst thing ever.

Maybe I should come up with a ranking of my own. I’d love readers to tell me what I should l rank at We can do this too, Jay. Just you watch.

Follow my blog all day, every day by bookmarking

Follow all the Post’s Education news & blogs on our Facebook fan page, the "PostSchools" feed on Twitter or our Education home page at

By Valerie Strauss  | February 1, 2010; 12:24 PM ET
Categories:  High School, Standardized Tests  | Tags:  challenge index  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: Willingham: 'Race to Top' a doomed bribery scheme
Next: College quotas for high schools?


Look, I think Jay is a tool as well, but your grasping for notice by him and looking for ideas on how to start your own ranking is....a bit desperate.

Unless I missed the snark in your post. If so, well played.

Posted by: popopo | February 1, 2010 1:02 PM | Report abuse

You're right - it is very difficult to determine if schools are doing a good job, but we could do a better job gathering data on school success.

Here are my suggestions:

1) Index measures like AP/IB participation and test scores to measures that accomodate for relative advantage such as %FARMS, ESL, student turnover, or Average Household Income for the school district. I attended one of the local high schools that always scores among the top and for which parents pay a mint in housing -- In that district the teacher's main job was to fend off over-involved parents and make sure that the kids showed up for class and had a pencil in their hands. The kids were so well prepared by their overall environment they could nearly have taught themselves. I believe that without taking relative advantage into account we simply perpetuate stereotypes about "good" and "bad" schools/school districts.

2) There should be a national, standardized test that covers basic to advanced math, science and literacy and is administered every year. In a common test like this, 9th grade students might only be expected to be 20% proficient but by 12th grade they might be expected to be 80% proficient. By testing the same knowledge and skills each year we could see if schools were improving their students' average knowledge and skills from year to year or not - and adequately use that data to inform our understanding of school performance. Using a method like this a school that improves students' results from 10% to 70% proficient would be recognized as doing a better job than one that starts off with a group of students who are 40% proficient and moves them to 80% proficient.

3) Imagine students & parents are their school's consumers. Poll students, parents and alumni to determine if the school is meeting (or met) their expectations. Are they getting what they want out of the school? Did the school meet their needs? Different populations have different demands and expectations, and the school not only has to meet some objective measures, but should also be providing the services and opportunities that students and parents want. -- These could be grouped into categories (e.g. access to teachers and staff, afterschool opportunities, athletics, access to supplemental teaching/resources, support for post-secondary opportunities, etc...) and scored for both importance and quality of service to tease out how well schools are meeting their population's needs.

Posted by: DCLocal20 | February 1, 2010 1:18 PM | Report abuse

Well said Valerie.

Too many other variables, such as those you've mentioned, are missing relating to Mathews Challenge Index and school rankings.

Hopefully surrounding school jurisdictions, Superintendants and School Boards do not use the Challenge Index list as their measure in determining quality education assurance.

Posted by: TwoSons | February 1, 2010 1:41 PM | Report abuse

The Challenge Index is yet another example of certain people in the media oversimplifying an issue to produce a sound-bite that gets them noticed and makes even more money for their company. (Unless they have the Wahington Post management to work for - desperately seeking new revenues while shrinking print editions.)
We all know that, however the Index might have been "intended", this is not how it is received. Most people go to their knee-jerk reaction of "this one is good, that one is bad" to make pronouncements and push their children into more crazy schedules. So much home-buying and gossip around the pool is driven by this stuff.

Oversimplifications like the Index do more damage than help in getting parents and the public to delve deeply into the problems of public educaton in America. This also pushes the politicians to continue the mess and turn it into law such as NCLB and Race-till-we-flop (from lack of understanding).
Until the working classroom teachers are solicited and fully involved in the research and decision-making on changing education, the wheels will continue to spin.

Posted by: 1bnthrdntht | February 1, 2010 2:29 PM | Report abuse

The Challenge Index is a horrible index and a complete joke. Jay WANTS people to put unprepared students in AP/IB classes, therefore he does, in fact, encourage schools to game the system.

And the problem isn't hypercompetitive parents looking to push their kids, but close to illiterate kids forced into classes they don't understand and don't care about, and then spending tons of taxpayer money for these students to take tests they will fail.

Posted by: Cal_Lanier | February 1, 2010 11:40 PM | Report abuse

Jay's Index is wonderful. It shines a spotlight on the abusive practice of schools denying their students the chance to challenge themselves. The gains in life from taking on a challenge - and the ultimate grade in the course is immaterial if the challenge was taken seriously and the course work done - are undeniable and very very real.
People need to look at what success really is - and it ain't measured in or predicted by GPA. It's the ability to handle life's challenges, and nothing - NOTHING - better prepares one for meeting life's challenges than having met other challenges before.
Jay's Index reflects this creator of success. Nothing more. If your child is already meeting challenges, than ignore Jay's Index. It's not for you. If your community's students are already challenged, ignore Jay's Index. It's not for you either. But if your children or your community's children aren't being as successful in life as you would think they could be, look to Jay's Index for one possible reason.
Jay's Index is immensely powerful in doing what he intended it to do - showing, in a relative sense, how well a school prepares its students for life. Nothing more, but most certainly nothing less.

Posted by: LoveIB | February 2, 2010 8:14 AM | Report abuse

The real measure of the quality of an education? Is the recipient capable of functioning as a self sustaining adult? Is he congizent of the world in around him and prepared to contribute to the general welfare of the community? Does her formal K12 education serve as a foundation on which life long learning can be built whether that means higher education, or on the job training, or simply personal enrichment? Unfortunately, the quality of life cannot be reduced to data points and that lies at the heart of our ongoing debate about what consititues a quality education. When pundits play with numbers, they sometimes forget those numbers respresent real people. When Mr. Mathews ranks schools on such limited information, I wonder if he has fully considered the potential repercussions on the students, the teachers and communities he so arbitrarily judges.

Posted by: susangraham | February 2, 2010 5:43 PM | Report abuse

The comments to this entry are closed.

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company