Network News

X My Profile
View More Activity

Posted at 5:00 AM ET, 12/22/2010

The lowdown on standardized tests and how they are scored

By Valerie Strauss

You wouldn’t be alone if you thought that the companies that score standardized tests have machines that can whip out the results in no time.

But they don’t, at least not for tests with questions that require a written response.
Instead, they hire tens of thousands of temporary workers, who are paid by the hour (often about $11 to $13 per), to decipher the answers of millions of kids from third to twelfth grade. They are often poorly trained and overworked.

That’s the assessment from one of those workers, Dan DiMaggio, who explains in detail his experiences in an article called “The Loneliness of the Long-Distance Test Scorer,” which you can find here.

His conclusion: “Why would people in their right minds want to leave educational assessment in the hands of poorly trained, overworked, low-paid temps, working for companies interested only in cranking out acceptable numbers and improving their bottom line? Though the odds might seem slim, our collective goal, as students, teachers, parents—and even test scorers—should be to liberate education from this farcical numbers game."

In the article, DiMaggio refers to a book by Todd Farley called “Making the Grades: My Misadventures in the Standardized Testing Industry,” an indictment of the industry based on Farley’s 14 years in the industry.

Here’s a new, somewhat satirical piece about standardized tests and scoring by Farley, first published on The Huffington Post.

By Todd Farley
Like the maddeningly successful author Diane Ravitch, I, too, have changed my mind about No Child Left Behind. Unlike the estimable Ravitch, however -- whose recent bestseller argues in exhausting detail against the very accountability measures that Ravitch long championed -- in the great testing debate I’ve gone from "con" to "pro."

Since 1994, when I first got hired as a lowly temp for measly wages to spend mere seconds glancing at and scoring standardized tests, until the release of my non-bestselling book last fall, I had steadfastly believed that large-scale assessment was a lame measure of student learning that really only benefited the multinational corporations paid millions upon millions upon millions of dollars to write and score the tests.

I began to see the error of my ways last Thanksgiving, however, just as soon as my huge son popped from his mother’s womb, keening and wailing, demanding massive amounts of food, a closet full of clothing, and the assistance of various costly household staff (baby-sitter, music teacher, test-prep tutor, etc.). Only then, as my little boy first began his mantra of "more, more, more," did I finally see standardized testing for what it really is: a growth industry. In these times of economic recession, it was a lesson I didn’t need to learn twice.

Since that educational epiphany, the benefits of standardized testing have become embarrassingly obvious to me, starting with the fact the industry has proven to be a jobs program virtually unmatched since FDR’s WPA.

There’s pretty much no one who can’t get a job in testing, whether it’s as one of the tens of thousands of temps hired each year to score student responses to tests or as one of the teacher/ex-teacher/once-knew-a-teachers hired to write them. Because of the massive influx of money swamping the testing industry due to President Obama’s Race to the Top, anyone who taught school or went to school or even drove past a school is eligible for work in the business.

Consider that in the last year I’ve seen people hired back to testing companies who had been run off in shame not long before.

I’ve watched people fired from one testing company immediately getting rehired by another.

I’ve witnessed tiny, mom & pop test-development vendors celebrating their first contract by immediately posting job listings on Craigslist, hoping to find someone, anyone, out there on the great, big Internets that might be able to help them write "rigorous," national tests.

Even me -- a guy who a year ago was spouting virulent anti-testing rhetoric on NPR and whose "down-with-standardized-testing!" editorials were gracing the pages of The New York Times, Christian Science Monitor, Education Week, etc... has been offered absolution, and today I earn fat paychecks from more than one of the industry’s stalwart companies. (No, I have no shame.)

Forgiveness, obviously, is one of the industry’s strengths, and once hired testing industry employees and companies both enjoy the sort of tenure that would make a teachers’ union proud -- those jobs and those contracts pretty much can’t be lost.

For instance, in the last year I’ve seen a test development company fired for poor quality work nearly immediately rehired with a contract four times as large.

I’ve watched multinational corporations like Pearson Education get dressed down in U.S. Department of Education audits and fined by their customers for shoddy work and then be awarded contracts worth hundreds of millions of dollars.

I’ve witnessed a state like Tennessee win a half a billion dollars in federal education money even after another audit pointed out scoring inequities on that state’s tests. Yup, the testing industry today is like that Cole Porter song "Anything Goes," which is just what an economy as troubled as our desperately needs.

Standardized testing’s other most redeeming characteristic may be a surprising one, but the industry has proven to be really good at recycling.

Just as an employee found wanting at one company can simply get a job at another (or can start his/her own lucrative consulting company), so can test questions and/or whole tests be used again and again and again. Test questions good enough to be used on a Chicago test can be used again on a New York City test, just as a complete test sold for use in California can be sold again for use in Texas.

It’s happening even with the exalted Common Core Standards, or CCS, those educational benchmarks that many people (Messieurs Obama and Duncan included) believe will save this country’s sorry educational system: Just the other day I saw a testing company advertising for people on Craigslist to re-align the company’s millions of existing test questions to those almighty new standards.

Even though those millions of test questions in the company’s database had been written to apparently crappy and definitely passé state standards, that innovative testing company was making them magically new simply by draping them in new clothes, linking those items -- absolutely unchanged -- to the CCS. That clever company was saving an immeasurable amount of time and effort by recycling those old items instead of writing new ones, just the sort of reuse of materials that I’m sure would make Al Gore proud (not to mention Gore’s minion, Davis Guggenheim).

While my many previous concerns about the efficacy of standardized testing have not gone away, the primacy of those concerns has been supplanted by the fact my son has an enormous appetite. In my view, today the standardized testing industry is like one of those phone booths filled with whirling Race to the Top millions, and the only smart thing for me to do as a husband and father is to grab as much as I can. Like Diane Ravitch, I’ve seen how wrong I used to be.

I am reformer! Hear me roar.


Follow my blog every day by bookmarking And for admissions advice, college news and links to campus papers, please check out our Higher Education page at Bookmark it!

By Valerie Strauss  | December 22, 2010; 5:00 AM ET
Categories:  Guest Bloggers, Standardized Tests  | Tags:  common core standards, how tests are scored, no child left behind, race to the top, standardized test industry, standardized test scoring, standardized tests, test industry, test scoring  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: The history of Rudolph the Red-Nosed Reindeer
Next: Bring back shop class


Step back for a moment and think about who are the primary customers for these testing companies? State governments. These test companies want a piece of that somewhat easy and steady tax-dollar pie, thus have built a whole industry around it. I'm not mad at them, though; it's up to us customers to demand a better product, which edu-bureaucrats have such a hard time doing.
Even further, there's enough expertise within a school or a school district, combined with today's technology, that schools/school districts could write and score their own tests; we could all but drive these folks out of business, but the edu-bureaucrats who feed off this incestuous relationship would fight to the death like a Klingon to preserve their gravy train.

Posted by: pdexiii | December 22, 2010 6:25 AM | Report abuse

To think we could simply find "new" graders/reviewers is like thinking we can find "new" teachers. We do not have a second or third string, nor do we have farm clubs so we get people with experience. As to their value, that is another story, however we know we will see many again.

Before our train of thoughts jump the tracks, let's remind ourselves the real issue here is the testing. Testing is a relatively new device because our school system is not ALL that old. We are learning and developing programs that will tell us how we perform, be it against some standard or in competing against one another.

I am against abolishing tests. It serves nothing or anyone to "not know" how our students perform. It should be used as feedback. Feedback to the student on their abilities and knowledge, feedback to the teachers/professors, feedback to the parents, and more importantly feedback to the system.

Tweak if we must, but aboloshing is another way to hide failure OR success.

Posted by: jbeeler | December 22, 2010 7:44 AM | Report abuse

$11 to $13 an hour for grading standardized tests? Dang. I make only about $8 an hour for substitute teaching. Where do I sign up? :-)

Posted by: barbarachina | December 22, 2010 8:43 AM | Report abuse

Blame the idiots who whined about the inadequacy of multiple choice tests. They aren't inadequate at all, and short answers are absurd and expensive.

Posted by: Cal_Lanier | December 22, 2010 9:26 AM | Report abuse

New York "won" the RttT beauty contest. With the money, aside from new contracts with Pearson, SED hired 25 new bureaucrats, and BOCES gets a 75% cut of the money each district gets. In my rural Upstate district, we'll end up with about $3.00 per kid which will more than be gobbled up by the testing costs that the state is passing down to local districts including a proposal to charge each district $5.73 per pupil to supply Regents exams at the high school level. Great job, Secretary Duncan!

Posted by: buckbuck11 | December 22, 2010 10:54 AM | Report abuse

I just looked at Florida's race to the top website and the results of their planning so far is a list of requests to vendors to apply for the hundreds of millions of dollars the state is getting. There is nothing there about schools actually receiving any of these funds directly. We have been cut out and the private companies will get this money and many of them will waste it because their primary goal is not to educate children but to make a buck.

Posted by: kmlisle | December 22, 2010 11:23 AM | Report abuse

Oh, Valerie, how I agree with the article posted here. But one thing alarms me. If the tea-partiers do some checking (and they will), they will find that DiMaggio's article was originally posted in Monthly Review, which since 1949 has proudly and boldly associated itself with socialist views. I'm a liberal, yet not as far left as a socialist, and I can only imagine what a group of Dick Armey-supported tea-partiers, as well-funded and media-savvy (and ruthless) as they are, will do to you and your efforts, and your column, should you keep posting from far-left sources. Let me be clear: I agree with Mr. Dimaggio, and have no doubt that what he says is true. But the right will demonize both he and you, and the wonderful credibility that you have established could be lost. I would advise caution in the future, even as you continue to make so many valid points.

Posted by: rtinindiana | December 22, 2010 12:29 PM | Report abuse

This is a great article. Hopefully Congress will read it.

Posted by: jlp19 | December 22, 2010 1:35 PM | Report abuse

Newsweek ought to read this also.

Posted by: jlp19 | December 22, 2010 1:37 PM | Report abuse

Valerie, this article comes across to me as a pretty incoherent laundry list of complaints. It leaves the question, what realistic steps would the author recommend to improve the situation?

There is the question of whether so much testing is advisable, and I agree that it's not. However, some testing is essential, so the problem is how that should be done. Multiple-choice tests have some major benefits for large-scale standardized testing, but formats such as short-answer or essay writing are needed as well. There have been impressive developments in computer scoring of these, but there are also strong arguments that human rating is also needed. Todd Farley doesn't really address how he would go about improving this, he simply takes cheap shots. What would be interesting would be to hear from test developers about why they make the decisions they do. Test development is an endless series of compromises, with cost and practicality unavoidable constraints on what is possible. How would Farley address these constraints?

Posted by: Trev1 | December 22, 2010 10:21 PM | Report abuse

Children are trained to produce writing that fits the test rubric, but nothing beyond that. They learn a formula consisting of a set number of paragraphs with set numbers of sentences and descriptive words. Then the scorers literally count whether the writing fits the rubric. It doesn't matter if the writing is garbage, as long as it fits the rubric. In many schools, the entire elementary school writing experience is directed towards teaching kids to write one kind of paper that can be scored by chimps.

One year, a lot of the the 4th graders at my GT magnet school (this was in Texas) had sub-par writing scores. It didn't make sense because they were all talented, skilled writers. The principal was able to get copies of their essays back from the state so the teachers could figure out what went wrong. The problem, as it turned out,was that they didn't respond to the stupid prompt, which was something like "Pretend you found a baby dinosaur. How would you care for it?" They got too creative and went off topic, so their scores were lowered. Of course, the teachers were careful to teach the kids how to write stupid for the stupid test in subsequent years.

These writing tests actively discourage the teaching of good writing. Children deserve better.

Posted by: aed3 | December 23, 2010 10:45 AM | Report abuse

I'm a former middle-school teacher and local playwright in Jacksonville, Florida. Sadly, nothing in your blog is news to me -- I wrote and produced an entire play about it! Read it at:

Posted by: DisgruntledTeacher | December 23, 2010 5:40 PM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.

characters remaining

RSS Feed
Subscribe to The Post

© 2010 The Washington Post Company