A hard look at education research
Florida’s public school system is frequently held up as an exemplary example of how specific school reforms -- vouchers, charter schools, high-stakes standardized tests, etc. -- can improve student achievement.
I recently wrote about the report -- which praised the business-driven, pro-choice reforms -- and said they should be viewed as a model for the country. (The Dec. 2 post was headlined, “Is progress in Florida schools exaggerated?”) It was written by Matthew Ladner and Lindsey Burke for the Heritage Foundation, and entitled “Closing the Racial Achievement Gap: Learning from Florida’s Reforms.”
The report was critiqued by Madhabi Chatterji of Teachers College at Columbia University, who questioned its methodology and conclusions.
Ladner and Burke sent me the following letter about the post, and after that is a response from Chatterji. I am posting it because I think the discussion reveals the controversial nature of much of “education research.”
For more on Florida's decade-long reforms and whether they really are as successful as claimed, you can read another post, a conversation I had with Professor Sherman Dorn of the University of South Florida, who has spent years researching and writing about public education in the Sunshine State.
Dear Ms. Strauss,
[The Dec. 2] Answer Sheet does a good job of summarizing Madhabi Chatterji’s critique of our study, Closing the Racial Achievement Gap: Learning From Florida’s Reforms. We wish you had given us the opportunity to respond to the questions Chatterji raises. We will do so now in this letter.
To review: Our study concluded that education reforms implemented under then-Governor Jeb Bush have produced significant academic gains, particularly among minority students. Florida’s Hispanic students, for example, are now outperforming or tied with the overall reading average for all students in 31 states.
The change averse may wish to quibble over the details, or agonize over just what reform did how much of what. But the bottom line is that in 1998, the year before Bush instituted his reforms, 47 percent of Florida 4th graders scored “Below Basic” on the NAEP [National Assessment of Educational Progress] 4th grade reading test. In 2009, that number was down to 27 percent. That is a 42% decline in the 4th grade illiteracy rate in the span of a decade.
Dr. Chatterji suggests that Florida’s policy barring social promotion of 3rd graders artificially inflates Florida’s 4th grade reading scores. Ironically, she criticizes our paper for not conducting a literature review even as she fails to do so herself. Had she reviewed the literature, she would have found that former Heritage Foundation Senior Analyst Dan Lips and Ladner had already addressed this, her main claim, over a year ago in the pages of Education Next.
The reality of the Florida policy reforms is far more complex and positive than Dr. Chatterji would have you believe:
* Florida’s 4th grade reading scores improved almost a full grade level between 1998 (when Bush began implementing portions of his reform agenda) and 2002. The 3rd grade retention policy had no chance to impact these scores; it was not yet in place.
* The maximum number of children retained--16% of all 3rd graders--came in the first year of the policy. The total number of students retained declined by more than 50% between 2002-03 and 2008-09. Chatterji’s own data show massive declines in the percentages of Florida minority students who ever wound up as illiterate 3rd graders in the first place.
* If retention were driving Florida’s post-2002 improvement as claimed, we should have expected to see Florida’s NAEP scores peak, and then decline when the number of students being retained substantially declined. Instead, what we find is that scores continued to improve even as the number of students being retained fell substantially.
* Chatterji attributes Florida’s improved scores to aging (retained students are a year older when they take the NAEP). Two statistical analyses of the 3rd grade retention policy compared students retained to two very similar groups of students: those who barely scored high enough to avoid retention, and those who scored low enough but avoided retention through an exemption. These analyses revealed that the retained kids outperformed these two control groups after one year, and by an even wider margin after two years.
* Florida lawmakers also created a mid-year promotion policy, retaining children only until they achieve an FCAT 2 [Florida Comprehensive Assessment Test] on reading with multiple attempts, whereupon they rejoin their cohort. In other words, many of the "retained" children now rejoin their classmates after improving their reading skills.
Dr. Chatterji also argues that other policies such as class size reduction and preschool vouchers—both enacted in Florida by ballot initiative—may have caused the improved scores. There is little or no reason to believe these policies helped improve Florida’s reading scores.
* There is a five year delay between attending preschool and reaching the 4th grade. The preschool program was passed in 2002, but wasn’t open to all 5-year-olds until 2005. The earliest year it could have an impact on 4h grade reading scores will be 2011.
* Florida’s class size amendment, passed in 2002, had a very slow implementation. Any effects from this measure would have shown up only in the later years, if at all. Moreover, a review of the literature shows that the empirical evidence on class size reduction is overwhelmingly negative.
The bottom line is that Florida’s education reform model, particularly the retention policy, incentivized schools to place a greater focus on early childhood literacy- in grades K, 1, 2, and 3. That is why the number of students scoring low enough to be retained in the first place has been cut in half, and Florida’s NAEP scores have improved.
As for us, we recommend that policymakers look to Florida as an example of successful education reform with its proven mixture of transparency, accountability and parental choice.
Matthew Ladner, Ph.D.
Vice President of Research
The Goldwater Institute
Lindsey M. Burke
Education Policy Analyst
The Heritage Foundation.
A response to that letter from Madhabi Chatterji:
1. The main problem with the Heritage Foundation report, "Closing the Achievement Gap...” by Ladner and Burke is that, using standard conventions of social science, the authors have not conducted a research study at all.
What they have done is point to selected, superficial results from the fourth grade NAEP reading test reports for the nation and a few selected states (Florida included). Those results look good (the Florida story is less impressive if one looks at NAEP scores beyond 4th grade reading, however). They then point to differences in reading means (averages) for selected ethnic sub-groups.
Finally, they describe some education reform policies from Florida that they favor, and they argue that other states will have similar test results if they adopt those policies. As noted, their selected reading test results for Florida look impressive on the surface.
However, the results do not tell us much about which education reforms led to them nor do we know how they were implemented. An actual research study would attempt to explore all possible causes, would attempt to control for as many factors as possible, and would note (the very large) limitations. Because this report does none of those things, it is merely an advocacy statement or a political position. It is certainly not a research study that permits cause-and-effect conclusions.
2. If the Heritage group or other researchers want to make cause-and-effect claims about Florida’s reforms, they will need appropriately gathered data that allow them to understand better whether Florida’s results are in fact improving more so than other states, after including appropriate controls and measures that help them link particular reform measures to student achievement.
They will need to demonstrate that the observed gains are attributable to the interventions they espouse, by ruling out alternative explanations, including effects due to the changed composition of student cohorts who are retained one or more times in a grade.They should also look to see if other states that have already adopted similar reforms are doing well. This is straight forward social science(not "quibbling", as Dr. Ladner suggests in his response). Ignoring sensible use of study designs and research methods is not at all desirable if we want to avoid pursuing misguided policies and associated expenditure of monetary and human resources.
3. As examples of alternative explanations to test score increases outside the cohort effect for grade retention, Florida’s literacy programs (“Florida Writes” and literacy coaching program), known to be strong, may be main factors causing reading gains.
Currently available research shows a strong relationship between reading comprehension and writing ability. But the Heritage authors never address the literacy programs.
Another alternative explanation could be the class size reduction reforms in the state. Even the grade retention programs in Florida that the report advocates apparently include a strong support element (e.g., students given learning supports over the summer) that may account for benefits even while the grade retention itself is harmful.
My point is merely that properly conducted investigations are called for to support their claims. There is no way for a reader to tell from the report’s presentations of data whether all, some, or none of the Florida efforts have actually helped students improve their achievement.
4. The Heritage authors also say that I overlooked their article in the Hoover Institution journal Education Next where they briefly addressed my concerns about grade retained cohort effects.
I did, in fact, read that article, but I did not find a serious treatment of the issue. Their short argument simply said that the retention cohort effects couldn’t possibly be accounting for everything seen in the results. That is a big claim that needs to be tested. What’s clear, however, is that any study of Florida should begin by taking the cohort effect (the movement of children between cohorts that is caused by the grade retention) into account and then go from there. Even if the Heritage authors are correct that this cannot account for everything, it certainly accounts for a substantial amount, and there is no sensible reason whatsoever for a researcher to ignore it.
5. The note that the authors of the Heritage report sent you includes a variety of other, subordinate contentions. For instance, the authors point to a study done a year or two ago concluding that the Florida class size reduction effort has been ineffective. But that study was reviewed by Jeremy Finn and found to be flawed.
6. In summary, the authors’ reactions to my review of the Heritage Foundation report indicate a lack of attention to study design factors and alack of value for research-based evidence for informing policy-making. This mindset is troublesome from where I stand. We should expect more from researchers at a time when the federal government is calling for evidence-based policies in education.
Follow my blog every day by bookmarking washingtonpost.com/answersheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page at washingtonpost.com/higher-ed Bookmark it!
| December 28, 2010; 10:38 AM ET
Categories: Alfie Kohn, Research, School turnarounds/reform | Tags: education research, florida school reform, florida schools, heritage foundation, school reform
Save & Share: Previous: Ravitch takes stock of education in 2010
Next: Can you correct the errors found in this Virginia textbook?
Posted by: CalP | December 28, 2010 12:29 PM | Report abuse
Posted by: peonteacher | December 28, 2010 12:32 PM | Report abuse
Posted by: efavorite | December 28, 2010 12:44 PM | Report abuse
Posted by: jrlund1 | December 28, 2010 12:45 PM | Report abuse
Posted by: steve1231 | December 28, 2010 12:46 PM | Report abuse
Posted by: zukermand | December 28, 2010 12:54 PM | Report abuse
Posted by: pbassjbass | December 28, 2010 12:57 PM | Report abuse
Posted by: buckbuck11 | December 28, 2010 1:06 PM | Report abuse
Posted by: educationlover54 | December 28, 2010 2:07 PM | Report abuse
Posted by: educationlover54 | December 28, 2010 2:09 PM | Report abuse
Posted by: joshofstl1 | December 28, 2010 4:42 PM | Report abuse
Posted by: ananna | December 28, 2010 5:37 PM | Report abuse
Posted by: schoolrunner | December 29, 2010 10:53 AM | Report abuse
Posted by: Ladner665 | December 29, 2010 7:35 PM | Report abuse
Posted by: rjma1 | December 29, 2010 10:14 PM | Report abuse
Posted by: msw13 | December 29, 2010 11:21 PM | Report abuse
Posted by: DHume1 | December 30, 2010 1:31 AM | Report abuse
Posted by: Ladner665 | December 30, 2010 9:51 AM | Report abuse
Posted by: fgabi | December 31, 2010 12:19 AM | Report abuse
Posted by: DHume1 | December 31, 2010 3:08 AM | Report abuse
Posted by: Ladner665 | December 31, 2010 2:36 PM | Report abuse
Posted by: DHume1 | December 31, 2010 7:45 PM | Report abuse
The comments to this entry are closed.