Network News

X My Profile
View More Activity

Posted at 8:00 PM ET, 01/ 5/2011

How overextended test security hurts learning

By Jay Mathews

I think Arlington County, if you allow for its relatively small size, is pound for pound the best school district in the nation. I am biased because it was my first beat when I was a young reporter in the 1970s. When I returned to cover it again in the late 1990s, I found a level of leadership, instruction and commitment to improvement still far beyond what parents and students usually encounter.

But even a district as splendid as Arlington can fall into the bad habit of letting obsession with test security and high-tech classroom assessment get in the way of a parent’s desire to help her child learn.

Late last year, I wrote about schools that keep graded tests hidden from students and parents for reasons that don’t make much sense. Here we go again.

Arlington parent Sarah Goodell has a daughter in third grade. She knew the child was being given a county-designed standardized test every quarter to monitor student progress. She thought that was a good idea.

Citing information from Assistant Superintendent for Instruction Mark Johnston, county schools spokeswoman Linda Erdos said the assessment “is used to inform teachers about areas of content in which a student may need additional support or assistance. It also tells teachers where students are performing well, thereby allowing the teacher to move on to other areas of instruction. It is not used in determining a student’s grade.”

Goodell didn’t care about the grade. She just wanted to see how her kid was doing so she could be as helpful as possible at home. Yet, she said, “I was told by my child’s teacher and principal that I could not have a copy of her quarterly math assessment per Arlington County rules. The teacher and principal were sympathetic to this point but said their hands were tied.”

Parents all over the country have been telling me about similar refusals by their school districts. What did Arlington, usually so well-organized and responsive to families, say for itself?

“At this time,” Erdos said, “there is only one version of the assessment, and unfortunately, we don’t have the resources to develop multiple versions of the test. For that reason, staff does not want to compromise its reliability to provide us with information that is needed to inform instructional decisions that ensure that students get the instructional support they need.”

Hmm. I still don’t get it, and neither does Goodell. I can see why the county might be concerned about preserving the test’s reliability if it were the SAT or the state Standards of Learning test, where bad scores could have serious consequences for students. Goodell was asking to see a test with no bad consequences, particularly for third-graders such as her daughter. Few parents are ever going to try to see the test result. Goodell is not planning to post it on the Internet. She just wants to see the questions being used to determine how her daughter will be taught.

Johnston said that if a parent asks, she is welcome to come to the school to look at the questions her child missed and get a clear idea of why the answers were wrong. “What we don’t do is release the test or copy it for the parent to take home,” he said.

It appears Johnston has not made this policy clear to his schools, and it still doesn’t make sense. If showing the questions to a parent at school does not threaten test reliability, why is sending the parent a copy a problem?

This reluctance to let parents or students take graded tests home is spreading. “It limits the parents’ ability to help children learn from their mistakes,” Goodell said.

Educators regularly express concern about the lack of parental involvement in students’ education, yet here they stymie a mother who wants to help the school district do its job.

Read Jay's blog every day, and follow all of The Post's Education coverage on Twitter, Facebook and our Education Web page.

By Jay Mathews  | January 5, 2011; 8:00 PM ET
Categories:  Local Living  | Tags:  Arlington County assistant superintendent for instruction Mark Johnston, Linda Erdos, Sarah Goodell, keeping parents from seeing their children's graded exams, test security hurting learning  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   Del.icio.us   StumbleUpon   Technorati   Google Buzz   Previous: Effective reading program shelved, then amazingly reborn
Next: Making college the goal helps non-college dreams too

Comments

I can't fault Arlington for this (and you're right, they are a model school system). The focus on testing has school systems doing many things that are completely absurd. Creating these tests is time consuming and expensive. Sharing them will, no matter what you think, lead to them becoming invalid. Most parents would not care to misuse the test questions, but some would, regardless of the fact that the test is not graded and does not impact their child in any way.

You would be better off chastising Arlington for giving these tests (think of the class time taken up by such nonsense) than for not sharing them.

Posted by: Jenny04 | January 5, 2011 9:40 PM | Report abuse

Jay, I'm glad you wrote this because now I understand why so many people are confused about why "teaching to the test" or teaching the test (actual items) is wrong.

First of all, let me explain about two kinds of tests: teacher-made and standardized. Let's take the example of spelling. In a classroom test the teacher might teach twenty spelling words in one week and then give a test at the end of the week on those words. In this case it would be her duty to inform the parents or show them the actual test if asked. That's because the goal is to teach those twenty words so of course you'd want the parents to know what they are.

Now let's take a standardized test. During the course of a year or a period of time, the third grade child might be expected to learn how to spell 200 words. It would be too cumbersome to test all 200 words so the test presents a SAMPLE of those words; let's say 20. If the child gets most of the words correct it is probable that he knows most of the 200.

Now let's say Mrs. Goodell gets a copy of the test. She is sure to drill her daughter on those twenty words, and her daughter is likely to get all those words right, leading Mrs. Goodell to believe her daughter is good in spelling even though it's quite possible that her child only knows how to spell half of the 200 words. The same would go for every other subject. The test has now lost its validity.

There is another issue as well. If one parent gets a copy of the test, she is sure to share it with her friends. If enough children are drilled on the exact items, the test can no longer give reliable information to the school or the parents.

There is a fairly simple solution to this problem. The school district can make up a similar version of the test so parents can know what to expect. However, to give out the exact test would defeat the whole purpose of testing because the results would not be valid. Also, if parents are educated about the concepts of sampling, validity and reliability, I think they'll understand why a standardized test cannot be made public.

I applaud the Arlington County administrators for their integrity. In many schools in the country (and D.C. I suspect) administrators are encouraging, even pressuring teachers to drill the kids on the exact items on the test so they can then brag about the "miraculous" improvements.

Posted by: Linda/RetiredTeacher | January 5, 2011 9:59 PM | Report abuse

This business about testing demonstrates how the affluent children get better services in so many ways - big and small. The Arlington administrators know that their students will probably do fairly well on the test, so they are guarding its validity closely. They want the correct information to give parents and teachers. They are not afraid of the truth.

Districts with very poor children, on the other hand, ARE afraid. They are afraid of public humiliation and threats. The administrators are afraid for their jobs and the teachers are afraid they will be labeled in the newspaper as "least effective" based on test scores. For these reasons, and many others, these districts are often quite happy to let teachers, parents and anyone else look at the tests. The booklets sometimes sit in teachers' rooms a week before they are administered. They often go to principals' offices afterwards so they can "examine" them before they're turned in to the central office. So when the parents get the "results" they may or may not be accurate.

At one charter school in a very poor district, the children made "phenomenal" progress on the standardized tests, much higher than most of the other schools in the city, including the affluent ones. However, when the very same children took a different test (Iowa Test of Basic Skills) they scored low, just as low as the children in the traditional schools and much lower than the children in the higher-income schools. The operator of that charter "solved" the problem by not giving the Iowa test again. So when those children went home with their "miraculous" scores, the parents were likely being misled about their children's progress.

The parents in Arlington are probably getting valid information about the progress of their students. I should think most parents would appreciate that.

Posted by: Linda/RetiredTeacher | January 5, 2011 10:53 PM | Report abuse

Friends taking the class next year and younger siblings taking the class in the future can cause problems. Recreating testing materials is very time consuming.

Posted by: mimizhusband | January 5, 2011 11:24 PM | Report abuse

I have a suggestion - design a "sample test" where each question is of the type of each question on the real test (different numbers, word problem changed, etc, but the same concept). Let the parent know results by presenting them with a *sample* of the type of questions the student missed.

In my children's Algebra I class in 8th grade, not only did the teacher return the tests, but the students could gain extra points for incorrect answers by writing the problem, solving it correctly, stating what caused the mistake and stating what the student will do differently to avoid making the same mistake. I thought this was brilliant.

Posted by: awrosenthl | January 6, 2011 7:34 AM | Report abuse

First, having a single test shows laziness. A single test can be written differently to provide multiple tests. It also says there is myopic ideas in the system.

Second, adults CAN play part in this scenario. It's called the vote. If you do not like those in charge, change their address and place of employment.

Last, in defense of not providing the test I recall one schoolhouse testing program where tests were posted in the main lobby every test day.

The alternative to all this noise is to provide a synopsis of the test to parents and students. It doesn't require an answer for answer results. Knowing the child needs assistance in say "adding multiple numbers" would tell me at least, what the child needs to practice. If I knew my child needed help on the French Revolution, or even parts of the French Revolution, I could at least work with my child.

The biggest problem here seems to be communications between parents and teachers and administrators. Voting does create more conversations!

Posted by: jbeeler | January 6, 2011 8:13 AM | Report abuse

This nonsense also happens in DC. Kids 3rd grade and higher take several DC BAS tests as practice for the DC CAS test. Our school refuses to share the results with the parents. I'm not sure if this is the widespread policy.

Each of these practice tests represent several wasted days when real learning could occur.

Posted by: drmommy | January 6, 2011 8:36 AM | Report abuse

I would love to do this for parents, but my biggest issue is the darn copyright restrictions. For example, one source I use for test questions has this written on the front cover: "Teachers are permitted to download the materials and make copies to use with their students in a classroom setting only. To maintain the security of this exam, teachers should collect all materials after their administration and keep them in a secure location. Teachers may not redistribute the files electronically for any reason." This is from a College Board document provided to approved AP teachers through a secure website because for many years we complained that we can't use mutliple choice questions from old AP tests because The College Board sells those old tests online for $25. I would rather have a fight with parents over not allowing students to take the tests home than go to court against the College Board in a copyright infringement case when the CB's instructions are that explicit.

Posted by: Rob63 | January 6, 2011 9:07 AM | Report abuse

Excellent comments. The suggestions from Linda/RetiredTeacher and awrosenthl have real merit. If anyone knows of districts that do provide such test samples to parents, let me know.

Posted by: Jay Mathews | January 6, 2011 10:56 AM | Report abuse

Actually, Rob63, the courts have explicitly found that sharing results of standardized tests with the families of the children tested constitutes fair use, and does not trigger copyright violation. See this summary of the relevant case:

http://brandlaw.org/2010/01/copyrighted-standardized-tests-is-there-a-fair-use/

Posted by: richardcmiller | January 6, 2011 11:02 AM | Report abuse

On another note, if school systems are purchasing tests, even if the Dept of Ed in Washington, and not including ownership on purchase, they are wasting tax payer services and dollars. If I buy a car and decide I want to paint it another color, the manufacturer cannot stop me.

We need to "purchase" the test, not lease it.

Posted by: jbeeler | January 6, 2011 11:09 AM | Report abuse

First, having a single test shows laziness.


Who exactly is being lazy in your view ?

Posted by: mamoore1 | January 6, 2011 11:45 AM | Report abuse

Prince George's County PS does the same.
Today, elementary students (gr3-6) begin the quarterly Formative Assessment tests (FAST) in reading and math. These tests mimic the Maryland State Assessment (MSA), Maryland's NCLB test.
The tests are not shared with parents, and are treated with the security used with MSA.
Administrators and proctors (who are the staff in the school, or a hired sub-no Central admin staff) are required to sign the same form used with MSA, which notes sanctions for violating test security. Sanctions include losing teaching certificate and reimbursement of re-take costs.

Posted by: edlharris | January 6, 2011 12:26 PM | Report abuse

I think what someone wants is to have their cake and eat it to. You want a good test that gives accurate information. This particular test is a formative test. It's main purpose is to guide instruction. Sounds like it's working. The well intended parent is missing the whole point. If the content of the test was known, and how hard should it be for an adult to remember the content of a test aimed at 8 year olds the the test loses most of its value. Even if the district wanted to spend the time and money to develop different tests, ( remember that the whole value of this test is that it measures student learning for all the students district wide, not one teacher's classroom ) the reliability of the test would be diminished until it was established for each version. If you keep releasing versions that will never happen.

Posted by: mamoore1 | January 6, 2011 12:52 PM | Report abuse

mamoore1, the system. If the best that can be found in all those involved is a single test, then they are lazy for either not making additional tests or enforcing additional tests.

Posted by: jbeeler | January 6, 2011 1:46 PM | Report abuse

awrosenthl, I give similar tests throughtout the college class I teach. The tests they get weekly are samples of but not exactly as the final. By the time they get the final they are not only better prepared, but more knowledgable about the important issues. Great point.

Posted by: jbeeler | January 6, 2011 1:51 PM | Report abuse

It seems odd that Jay Mathews, who claims to be an expert on education, obviously doesn't understand anything about testing.

The purpose of testing is to measure the test-taker's knowledge of a body of information. For practical reasons, tests measure only a small but representative sample of the total body of information. When used properly, tests provide a reasonably accurate measure of the test-taker's knowledge of the total body of information.

It is good practice to give test-takers practice test questions prior to testing but not the actual test questions themselves. If test-takers study the actual test questions, rather than the total body of information, the test will provide an inaccurate measure of the test-taker's knowledge of the total body of information. Thus, test security is needed to ensure reliable and valid test results.


Posted by: DrFlyNomo | January 6, 2011 2:21 PM | Report abuse

Jay,

I agree totally with DrFlyNomo. It's important for you to understand testing before writing about it. The public needs to be educated about these tests and who better to do it than journalists?

Here's something I would love for you to do. I've suggested it before but here it goes again. Go to one of your schools with very poor children and "miraculous" test scores. See if you can find out if the teachers are drilling the kids on the exact test items. Ask the school if they'd be willing to give the kids another test (unseen of course). If not, ask if you can assign an in-class composition to sixth graders. You'll find out a lot by reading these compositions. If children are truly at grade level, their writing will demonstrate it.

Do I sound cynical? Sorry, but I've seen a lot.

Posted by: Linda/RetiredTeacher | January 6, 2011 4:47 PM | Report abuse

Jay, given how often issues about testing come up here, why not interview some psychometricians working in test development about these issues? You may find that there are sound reasons for the decisions they make.

Posted by: Trev1 | January 6, 2011 5:13 PM | Report abuse

Hi Jay, I am a math teacher in Arlington, and we have spoken via email before. With all due respect, you are off base on this call. These benchmark tests are made by teachers and run through a rigorous vetting process. They take enormous amounts of time to create. Each question aligns with one or more standards of learning (included in the key so that teachers and parents can quickly refer to the bigger picture). In middle school, the benchmark tests are multiple choice, which makes them very hard to write (the wrong answers need to mimic student misconceptions if the test is goint to be valid). Students and I review the tests after taking them, and their results inform my instruction. In my classroom, parents get scores, as I include them on the student grade record. I think Ms. Goodell is fighting the wrong battle. She has access to the test; that is more than fair and transparent.

Linda Allen

Posted by: grandfam | January 6, 2011 8:33 PM | Report abuse

Jay - you wrote, "I can see why the county might be concerned about preserving the test’s reliability if it were the SAT or the state Standards of Learning test, where bad scores could have serious consequences for students."

Do you really not understand that reliability is an important feature of ALL tests/assessments or else the test is pointless? I can say without hesitation that there are parents (particularly in this high-achieving/ high-pressure area) that don't want their kids to fail at anything - even tests without "serious consequences." Releasing these tests would make the results essentially meaningless and the students whose learning needs may be hidden will go along without getting the help they need.

Posted by: TerpzFan | January 7, 2011 7:20 AM | Report abuse

Jay, parents already have access to the material that their students will be tested on. All local school districts post their Program of Studies on line (Arlington's is at http://www.apsva.us/1540108115415800/site/default.asp). A parent just has to go to the appropriate grade level and class and it will tell you what are the standards being taught. Many school districts are also posting pacing guides which show what subject is being taught at a specific period of time. In addition, I would suggest to Ms. Goodell to ask for her daughter's past in-class tests back (if she hasn't already received them) since surely the teacher is assessing the material likely to be on this quarterly assessment. I don't think the school districts are purposely keeping the actual tests from parents because they don't want parental involvement. It is more for the point of not having to create a new bank of questions each and every year. I am not sure if this quarterly test is done on computers, but if it was, a test bank of say 100 questions could be easily randomized so that the students don't see the same questions year after year. Even if it is not computerized, each paper test could be slightly different from the year before by ensuring you have more questions in the test bank than appear on the test. In any case, parents have tools at their disposal to know what subjects are being tested and can prepare/practice at home with questions of their own creation if that is what they want to do. I have done this for both of my kids (one middle school, one elementary school) and have had no need to look at quarterly tests because I already know what they are testing on prior to the test being given.

Posted by: APforyou | January 7, 2011 8:47 AM | Report abuse

Our assessments are web-based giving our teachers results 24 hours after a student takes the assesment. I would think the skills a child needs to develop is more important than the actual questions. If we are asked, we provide a report of results based on skills, concepts and other learnings. I would want to know a student is having difficulty with two-digit addition rather than they missed question #6. I would suggest Arlington create a report that offers this data for both teachers and parents.
--Supt. of rural NYS district.

Posted by: rsquier | January 7, 2011 12:32 PM | Report abuse

To "rsquier," Supt. of rural NYS district, Arlington already does exactly what you suggest: "I would suggest Arlington create a report that offers this data for both teachers and parents."

Posted by: grandfam | January 7, 2011 12:40 PM | Report abuse

More good comments:

For TerpzFan---I understand the reliability problem, but i have my doubts it is relevant in the case of low impact, largely overlooked testing like this. If you have some research based on tests like this that WERE shared with parents, I would buy yr view. But without that, as I said, I have big doubts.

For APforyou---It is nice to have the standards, and nice to have guidelines on the tests, but I think parents need--I certainly would need it as a reporter--the actual form of the questions the kid is missing, and even better the question itself, before I was confident that I knew what the problem was. It could be that the student understands the concept, but is confused by some aspect of the way the question is phrased. If I am going to learn to ride a bike, I have to do it on a bike, and not just watch somebody else ride a bike and explain it to me.

Posted by: Jay Mathews | January 7, 2011 1:37 PM | Report abuse

Jay, I think you're misunderstanding some important aspects of this issue.

The tests in question are not intended to be classroom quizzes, aimed at measuring achievement on specific things taught in class and guiding students and parents on what to review. Rather, they will have (or at least should have) been painstakingly developed to measure specific things that are good proxies of general development. Practical restrictions on test length mean that only a small sample of the infinite set of possible test items can be tested, so test developers pilot new tests, select the best performing items for the operational tests, and discard the poorly performing items. If teachers, students, or parents start using such tests as a guide as to what to study, the result is "curriculum narrowing". Test developers warn against basing curricula narrowly on tests because, as explained above, test content cannot cover all curriculum content. The technical term for this is test "washback" or "backwash".

It's disingenuous to demand that test developers prove that relaxing security would affect the validity of their test results. To do this would be extremely difficult and ethically dubious. It would require large scale administration of parallel test forms under operational conditions solely for the purpose of answering an academic research question, not to provide direct benefit to the students. It would require misleading the students and parents as to the purpose of the test (in order to ensure that the perception of the test stakes in the research administration and operational administrations could be assumed to be the same).

Just step back and think about this whole issue. The test developers and teachers in this case want to improve the education of these children. They developed a test instrument to measure their learning. Rather than go to the enormous expense of developing and validating multiple equated tests, they chose to reuse a single test form and for commonsensical reasons wish to maintain test security. A single parent who is merely curious wants to use the results in a way that the test is not intended to be used and which will not be of any benefit to the child in question beyond satisfying curiosity. If large numbers of parents did the same, the test would lose validity and the school district would either have to abandon the assessment or divert resources from other areas to developing equated parallel forms. All this because a single parent was curious about how their child did on a test that has absolutely no consequences. Either way, the children in this school district would be worse off by following your suggestions.

Posted by: Trev1 | January 7, 2011 9:10 PM | Report abuse

I think more than anything else this strand has shown why the know-nothings who pretend to be experts in education yet have no training or experience have it so wrong, and are doing a lot of damage.

Posted by: mamoore1 | January 7, 2011 9:35 PM | Report abuse

I am a parent of 5 kids and I am far from a "know-nothing." I do think that a parent is entitled to see their child's standardized test scores, particularly if they are "weak" or didn't score well. Otherwise...why care? If a teacher won't release a test that isn't standarized and a grade is at stake, then I'd make a federal case out of it. The teacher may have created the exam, but the responses are the student's work product and should be available to the student and/or their parent(s).
I'm not a fan of standarized tests. I usually exempted my public school kids, which is a parental perogative in California, where I live. My two special ed kids tended do poorly and I thought that the testing was a considerable waste of time AND money. Lots of money. The "STAR" testing that is done in Cali can take up to four weeks of instruction time. Talk about distruption to the learning process.
My youngest goes to Catholic school and our diocese uses the ITBS tests for curriculum planning only. The closest thing to a "standardized" test my daughter will take is the high school placement exam when she is in grade 8. As for the SAT...attend community college for a minimum of two semesters and the need for the exam is moot. Actual transcripts for college level work is a far better measure for success at four year school than any exam.

Posted by: kodonivan | January 10, 2011 8:15 PM | Report abuse

I am a parent of 5 kids and I am far from a "know-nothing."

No, you are not. Perhaps there is a reason both of your kids are sped and performing poorly. Reality is harsh, but the first step.

Posted by: mamoore1 | January 10, 2011 10:07 PM | Report abuse

Jay says:
It could be that the student understands the concept, but is confused by some aspect of the way the question is phrased.

While this is entirely possible, the data generated from the testing should reveal that there is a problem with how well the question assesses understanding. That question should be revised.

Even if we assume that there is a question that little Suzie keeps getting wrong, even though she understands the concept, how would the parent help the child overcome this if she/he saw the test (which it sounds is possible in Arlington)? Would this parent "tutor to the test"? Teach strategies specific to that type of question so that lil'Suzie can do better on a formative assessment?

Posted by: limnetic792 | January 11, 2011 12:17 AM | Report abuse

Post a Comment

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.




characters remaining

 
 
RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company