Network News

X My Profile
View More Activity

Posted at 10:38 AM ET, 12/28/2010

A hard look at education research

By Valerie Strauss

Here is a back-and-forth between the authors of a Heritage Foundation report on Florida’s school reforms and the author of a critique of the report.

Florida’s public school system is frequently held up as an exemplary example of how specific school reforms -- vouchers, charter schools, high-stakes standardized tests, etc. -- can improve student achievement.

I recently wrote about the report -- which praised the business-driven, pro-choice reforms -- and said they should be viewed as a model for the country. (The Dec. 2 post was headlined, “Is progress in Florida schools exaggerated?”) It was written by Matthew Ladner and Lindsey Burke for the Heritage Foundation, and entitled “Closing the Racial Achievement Gap: Learning from Florida’s Reforms.

The report was critiqued by Madhabi Chatterji of Teachers College at Columbia University, who questioned its methodology and conclusions.

Ladner and Burke sent me the following letter about the post, and after that is a response from Chatterji. I am posting it because I think the discussion reveals the controversial nature of much of “education research.”

For more on Florida's decade-long reforms and whether they really are as successful as claimed, you can read another post, a conversation I had with Professor Sherman Dorn of the University of South Florida, who has spent years researching and writing about public education in the Sunshine State.

Dear Ms. Strauss,

[The Dec. 2] Answer Sheet does a good job of summarizing Madhabi Chatterji’s critique of our study, Closing the Racial Achievement Gap: Learning From Florida’s Reforms. We wish you had given us the opportunity to respond to the questions Chatterji raises. We will do so now in this letter.

To review: Our study concluded that education reforms implemented under then-Governor Jeb Bush have produced significant academic gains, particularly among minority students. Florida’s Hispanic students, for example, are now outperforming or tied with the overall reading average for all students in 31 states.

The change averse may wish to quibble over the details, or agonize over just what reform did how much of what. But the bottom line is that in 1998, the year before Bush instituted his reforms, 47 percent of Florida 4th graders scored “Below Basic” on the NAEP [National Assessment of Educational Progress] 4th grade reading test. In 2009, that number was down to 27 percent. That is a 42% decline in the 4th grade illiteracy rate in the span of a decade.

Dr. Chatterji suggests that Florida’s policy barring social promotion of 3rd graders artificially inflates Florida’s 4th grade reading scores. Ironically, she criticizes our paper for not conducting a literature review even as she fails to do so herself. Had she reviewed the literature, she would have found that former Heritage Foundation Senior Analyst Dan Lips and Ladner had already addressed this, her main claim, over a year ago in the pages of Education Next.

The reality of the Florida policy reforms is far more complex and positive than Dr. Chatterji would have you believe:

* Florida’s 4th grade reading scores improved almost a full grade level between 1998 (when Bush began implementing portions of his reform agenda) and 2002. The 3rd grade retention policy had no chance to impact these scores; it was not yet in place.

* The maximum number of children retained--16% of all 3rd graders--came in the first year of the policy. The total number of students retained declined by more than 50% between 2002-03 and 2008-09. Chatterji’s own data show massive declines in the percentages of Florida minority students who ever wound up as illiterate 3rd graders in the first place.

* If retention were driving Florida’s post-2002 improvement as claimed, we should have expected to see Florida’s NAEP scores peak, and then decline when the number of students being retained substantially declined. Instead, what we find is that scores continued to improve even as the number of students being retained fell substantially.

* Chatterji attributes Florida’s improved scores to aging (retained students are a year older when they take the NAEP). Two statistical analyses of the 3rd grade retention policy compared students retained to two very similar groups of students: those who barely scored high enough to avoid retention, and those who scored low enough but avoided retention through an exemption. These analyses revealed that the retained kids outperformed these two control groups after one year, and by an even wider margin after two years.

* Florida lawmakers also created a mid-year promotion policy, retaining children only until they achieve an FCAT 2 [Florida Comprehensive Assessment Test] on reading with multiple attempts, whereupon they rejoin their cohort. In other words, many of the "retained" children now rejoin their classmates after improving their reading skills.

Dr. Chatterji also argues that other policies such as class size reduction and preschool vouchers—both enacted in Florida by ballot initiative—may have caused the improved scores. There is little or no reason to believe these policies helped improve Florida’s reading scores.

* There is a five year delay between attending preschool and reaching the 4th grade. The preschool program was passed in 2002, but wasn’t open to all 5-year-olds until 2005. The earliest year it could have an impact on 4h grade reading scores will be 2011.

* Florida’s class size amendment, passed in 2002, had a very slow implementation. Any effects from this measure would have shown up only in the later years, if at all. Moreover, a review of the literature shows that the empirical evidence on class size reduction is overwhelmingly negative.

The bottom line is that Florida’s education reform model, particularly the retention policy, incentivized schools to place a greater focus on early childhood literacy- in grades K, 1, 2, and 3. That is why the number of students scoring low enough to be retained in the first place has been cut in half, and Florida’s NAEP scores have improved.

As for us, we recommend that policymakers look to Florida as an example of successful education reform with its proven mixture of transparency, accountability and parental choice.


Matthew Ladner, Ph.D.
Vice President of Research
The Goldwater Institute

Lindsey M. Burke
Education Policy Analyst
The Heritage Foundation.


A response to that letter from Madhabi Chatterji:

1. The main problem with the Heritage Foundation report, "Closing the Achievement Gap...” by Ladner and Burke is that, using standard conventions of social science, the authors have not conducted a research study at all.

What they have done is point to selected, superficial results from the fourth grade NAEP reading test reports for the nation and a few selected states (Florida included). Those results look good (the Florida story is less impressive if one looks at NAEP scores beyond 4th grade reading, however). They then point to differences in reading means (averages) for selected ethnic sub-groups.

Finally, they describe some education reform policies from Florida that they favor, and they argue that other states will have similar test results if they adopt those policies. As noted, their selected reading test results for Florida look impressive on the surface.

However, the results do not tell us much about which education reforms led to them nor do we know how they were implemented. An actual research study would attempt to explore all possible causes, would attempt to control for as many factors as possible, and would note (the very large) limitations. Because this report does none of those things, it is merely an advocacy statement or a political position. It is certainly not a research study that permits cause-and-effect conclusions.

2. If the Heritage group or other researchers want to make cause-and-effect claims about Florida’s reforms, they will need appropriately gathered data that allow them to understand better whether Florida’s results are in fact improving more so than other states, after including appropriate controls and measures that help them link particular reform measures to student achievement.

They will need to demonstrate that the observed gains are attributable to the interventions they espouse, by ruling out alternative explanations, including effects due to the changed composition of student cohorts who are retained one or more times in a grade.They should also look to see if other states that have already adopted similar reforms are doing well. This is straight forward social science(not "quibbling", as Dr. Ladner suggests in his response). Ignoring sensible use of study designs and research methods is not at all desirable if we want to avoid pursuing misguided policies and associated expenditure of monetary and human resources.

3. As examples of alternative explanations to test score increases outside the cohort effect for grade retention, Florida’s literacy programs (“Florida Writes” and literacy coaching program), known to be strong, may be main factors causing reading gains.

Currently available research shows a strong relationship between reading comprehension and writing ability. But the Heritage authors never address the literacy programs.

Another alternative explanation could be the class size reduction reforms in the state. Even the grade retention programs in Florida that the report advocates apparently include a strong support element (e.g., students given learning supports over the summer) that may account for benefits even while the grade retention itself is harmful.

My point is merely that properly conducted investigations are called for to support their claims. There is no way for a reader to tell from the report’s presentations of data whether all, some, or none of the Florida efforts have actually helped students improve their achievement.

4. The Heritage authors also say that I overlooked their article in the Hoover Institution journal Education Next where they briefly addressed my concerns about grade retained cohort effects.

I did, in fact, read that article, but I did not find a serious treatment of the issue. Their short argument simply said that the retention cohort effects couldn’t possibly be accounting for everything seen in the results. That is a big claim that needs to be tested. What’s clear, however, is that any study of Florida should begin by taking the cohort effect (the movement of children between cohorts that is caused by the grade retention) into account and then go from there. Even if the Heritage authors are correct that this cannot account for everything, it certainly accounts for a substantial amount, and there is no sensible reason whatsoever for a researcher to ignore it.

5. The note that the authors of the Heritage report sent you includes a variety of other, subordinate contentions. For instance, the authors point to a study done a year or two ago concluding that the Florida class size reduction effort has been ineffective. But that study was reviewed by Jeremy Finn and found to be flawed.

6. In summary, the authors’ reactions to my review of the Heritage Foundation report indicate a lack of attention to study design factors and alack of value for research-based evidence for informing policy-making. This mindset is troublesome from where I stand. We should expect more from researchers at a time when the federal government is calling for evidence-based policies in education.


Follow my blog every day by bookmarking And for admissions advice, college news and links to campus papers, please check out our Higher Education page at Bookmark it!

By Valerie Strauss  | December 28, 2010; 10:38 AM ET
Categories:  Alfie Kohn, Research, School turnarounds/reform  | Tags:  education research, florida school reform, florida schools, heritage foundation, school reform  
Save & Share:  Send E-mail   Facebook   Twitter   Digg   Yahoo Buzz   StumbleUpon   Technorati   Google Buzz   Previous: Ravitch takes stock of education in 2010
Next: Can you correct the errors found in this Virginia textbook?


I believe the more important issue about achievement in education should be centred around whether or not we are teaching and measuring the right things about education.

As the late Christie McCauliffe (teacher/astronaut) stated, when asked about what she did as a teacher, she responded" "I touch the future, I teach". Are we able to measure how students would perform in the future? In the past, pedagogy was satisfactory to deal with knowledge that was fairly static; however, the rate of change of knowledge has accelerated greatly, which suggests that even in elementary school, students should be educated, moreso, about how to learn, rather than what to learn.

The benefits of education are long term in nature, unless we can come up with examinations in the short term that are good predictors of future performance, we are, probably missing the boat in the evaluation of students and of teachers and schools. And this is part of the reason for the need of remediation in many instances.

Even, and especially, in education, we must prescribe according to the needs of the day, with our eyes on the future.

Posted by: CalP | December 28, 2010 12:29 PM | Report abuse

I doubt either of those 3 could teach a fish to swim.

Posted by: peonteacher | December 28, 2010 12:32 PM | Report abuse

Not having checked the details of this issue myself, all I can say is that Chatterji sounds like a social researcher speaking clearly.

In contrast, Ladner and Burke sound like slick operators, trying to lead people down the garden path with use of the word "quibble" as a major clue (hoping to minimize something that might be important). Another clue is that the link takes you to the home page of Education Next, but not to the article in question, suggesting the writers want to give the appearance of transparency, without actually providing it.

So now I’ve given clues to Burke and Ladner and people like that on how to avoid looking like charlatans, whether or not they are. I doubt they’ll change anything – there’s no need to. Most people won’t check very carefully – either for lack of interest or lack of time.

Perhaps Burke and Ladner would be concerned about the effects these kinds of actions could ultimately have on the children they seem to care about. Let’s hope so.

Posted by: efavorite | December 28, 2010 12:44 PM | Report abuse

This is an interesting discussion. I would find the Lander and Burke response more convincing if they took the issue less personally. I would find the Chatterji response more convincing if it added some things that we do know. The changes in performance seem very large, and promising. The role of science in public policy is often hindered by it's being taken up in a campaign style by advocates as well as by scientists themselves focusing on what is not completely known, rather than what is both known and useful. Still, I enjoyed the contrast of two cultures of science in public policy.

Posted by: jrlund1 | December 28, 2010 12:45 PM | Report abuse

Get with the game, Dr. Chatterji: You're not supposed to let the scientific method and logic get in the way of a good story (particularly when the story is ideological political spin).

P.S. Ladner's PhD is in Poltical Science, not Education. My hunch is that a PhD program in Political Science is not heavy in research methods or statistics, unlike a PhD (not EdD) program in Education. I would hazard to guess that a PhD in Political Science is heavy on spin and propaganda.

Posted by: steve1231 | December 28, 2010 12:46 PM | Report abuse

Conservative "science" in 2 sentences:

"The change averse may wish to quibble over the details, or agonize over just what reform did how much of what."

"The main problem with the Heritage Foundation report, "Closing the Achievement Gap...” by Ladner and Burke is that, using standard conventions of social science, the authors have not conducted a research study at all."

Our public discourse is a sick joke and you, Ms Strauss, could be more helpful.

Posted by: zukermand | December 28, 2010 12:54 PM | Report abuse

How do the Flordia results compare to Montegomery County, Maryland, which also claims to have closed the achievement gap?

Posted by: pbassjbass | December 28, 2010 12:57 PM | Report abuse

The condescension in the Ladner and Burke's statement, "the change averse may wish to quibble over the details..." Really? Anybody who challenges their evidence or conclusions is to be dismissed as "change averse?" Retention policies, class size initiatives not favored by the Heritage Foundation's desired narrative are minimized. I notice no mention is made of math scores or eighth grade scores on the NAEP? Make no mistake, Ladner and Burke are not researchers. They are propagandists.

Posted by: buckbuck11 | December 28, 2010 1:06 PM | Report abuse

Thank you, Madhabi Chatterji.

Posted by: educationlover54 | December 28, 2010 2:07 PM | Report abuse

"As examples of alternative explanations to test score increases outside the cohort effect for grade retention, Florida’s literacy programs (“Florida Writes” and literacy coaching program), known to be strong, may be main factors causing reading gains."

I am surprised the Heritage Foundation didn't take this into account.

Posted by: educationlover54 | December 28, 2010 2:09 PM | Report abuse

"The change averse may wish to quibble over the details, or agonize over just what reform did how much of what. But the bottom line is..."

Translation = rather than analyze what elements of Florida's policy works or try to control for variables, let's just keep doing it all because, gosh, it sure *seems* to be working pretty well. Oh, and if you actually try to sift out the good from the bad in Florida's policies, you're a change HATER.

There's a partisan think tank for you.

Posted by: joshofstl1 | December 28, 2010 4:42 PM | Report abuse

For the last seven years I have taught in Florida and this is not a state that anyone should be looking at as a model for educational reform.
The only people benefiting from the onslaught of testing are the corporations creating the tests and the politicians.The tests are skewed, the results are skewed and they change the game to keep making the state look like the reforms are working.
I've taught Elementary, Middle and High School and one thing that is very clear to me is that the students we retain in elementary often become socially maladjusted in middle school and drop out in high school.
Florida has one of the highest drop out rates in the country.
So before we look at the reforms shouldn't we be looking at the results?

Posted by: ananna | December 28, 2010 5:37 PM | Report abuse

Here's what's missing from this and so much of the "Education Research" discussion. How to assist teachers with students TO ENGAGE THEM. Regardless of the amazing curriculum or content, students who are not engaged will not learn. Many teachers struggle with reaching students and students (incorrectly) feel that teachers are also disengaged. Until we begin to address this more directly, we are omitting key questions.

Posted by: schoolrunner | December 29, 2010 10:53 AM | Report abuse

Ms. Strauss-

Thank you for posting both responses. You are correct that there is a cultural difference between academia and policy making at play.

Florida policymakers enacted multiple reforms at once in 1999, with other reforms coming on later. Chatterji complains that we fail to disentangle just how much improvement which policies caused. With so many policy changes happening simultaneously, this is impossible in a statistical sense.

Florida education reform did not unfold as an intention to treat random assignment study in a lab, but rather in the hurly-burly world of politics and policy implementation.

Chatterji, in essence, is holding us to an entirely unrealistic standard. If not, then I invite her to attempt to statistically untangle this glorious mess. Read your Campbell and Stanley first.

As an academic exercise, it could be jolly fun to try. As a practical matter- here is what we know: Florida's 4th grade functional illiteracy rate dropped from 47 percent in 1998 to 27 in 2009. That is a jaw-dropping improvement on the nation's most highly respected K-12 test.

Chatterji's recreation of the Haney 3rd grade retention claim is ultimately unpersuasive as a major factor, for reasons Burke and I cite above. In the following link, I note that Chatterji's own chart shows that the percentage of children scoring low enough to qualify for retention dropped by 38%. Something is obviously going right in Florida:

Policymakers wishing to address widespread childhood illiteracy should indeed take note of the research, but they cannot afford to be paralyzed. Let's assume for a moment that Chatterji is correct that mere aging due to the retention policy increases the "real" below basic rate in Florida literacy scores by 10 percent. This doesn't square with the evidence (retention rates substantially dropping, scores going up, regression discontinuity analysis) but let's roll with it.

This might be of academic interest, but would not be of much practical importance- the percentage of Florida students scoring below basic would move from around 27 percent to just below 30 percent. It was 47 percent in 1998, and policymakers in many states would kill for a mere 30%.

Fortunately, we don't need to know exactly which reform did what. As a practical matter, we can recommend that policymakers do all of them.

Why did Burke and I ignore the class size bit? Because the policy was implemented late in the game, long after the improvement started. Plus, we've read our Hanushek. What about preschool? Florida children using the preschool voucher program haven't aged enough to make the 4th grade sample yet.

Policymakers must make decisions based upon the available evidence, and must always live with some degree of uncertainty. The academic perspective is important but has limits. The only thing that is certain is the widespread failure of existing policy.


Posted by: Ladner665 | December 29, 2010 7:35 PM | Report abuse

Thanks Josh. More teaching, less testing. Get kids excited about reading and they will read better. I also trust my kids teachers assessment about how they are doing than any standardized test results.

Posted by: rjma1 | December 29, 2010 10:14 PM | Report abuse

Given that the Heritage Foundation has clearly-stated political goals, why would anyone treat their statements as anything but partisan spin? How hard is it to cherry pick random data points, while ignoring overall data trends?

Posted by: msw13 | December 29, 2010 11:21 PM | Report abuse


"The only thing that is certain is the widespread failure of existing policy."

According to your own words, everything anyone else is trying outside Florida's boundaries is a "widespread failure." Because you are obviously "committed to accurate research," I highly suggest that you be a little more circumspect in the future; if you do not, then you risk tainting the Gold-water well that you and the rest of your group drink from.

Posted by: DHume1 | December 30, 2010 1:31 AM | Report abuse


I did not say that everything anyone else is trying outside of Florida is failing, merely that academic failure is widespread. "Widespread failure" seems like a fairly measured way to describe a country where a third of the students score Below Basic in 4th grade reading and that is humiliated in international comparisons of academic achievement on a regular basis.

That does not mean that there aren't excellent public schools and great people working in them. My own children attend just such a school. What it does mean is that we have a very serious problem, and that the status-quo does not deserve some sort of enormous benefit of the doubt.

I also admire the NAEP gains demonstrated in DCPS over the last decade, and MA's reform effort as well. Florida's gains have been larger, but there are lessons to be learned from other jurisdictions as well.

Those of you who are skeptical of think-tanks have no need to trust me. You can look the data up for yourself:

Posted by: Ladner665 | December 30, 2010 9:51 AM | Report abuse

Data was designed to be manipulated. The Florida data that was reported was the data that the State of Florida wanted all the other states to see. I agree about the cherry picking.

Posted by: fgabi | December 31, 2010 12:19 AM | Report abuse


Here's why there are snags in your Goldwater sweater of studies:

1. It is obvious that you want your study to be taken seriously. If that is the case, then you should focus on what was proven, let it stand like Oxymandias's statue in the desert, and leave it to the wind and weather to see if it stands. However, you seem to take the opposite approach. You've tied enough "fairly measured" semantic and rhetorical anchors to it and your earlier post as a means to keep your inductive fallacies from sinking in the desert sand.

I've put the correct links to your study and your other earlier, suspiciously serious and similar studies here if anyone wants to judge for him or herself:

"Demography Defeated" Sept 2008

"On the Road to Excellence" June 2010

"Closing the Racial Achievement Gap" Sept 2010

2. You used the "status-quo" argument (I just cringed in my seat). I can tell you this--and I'm being very candid here--whenever I hear this argument, I think the person who said it is a moron. Here's why: About every school district in the US-- especially in inner-city districts--is trying to raise test scores. And almost every year they are trying something new. In the last twelve years alone, waves of reforms and pedagogical practice perspectives have been pounding on the doorsteps of each classroom across the nation. So, when people use the status-quo argument, it turns the whole nation of divergent educational practices and beliefs into a tight, little moronic box of "it is us versus them" so as to make your point seem as though you are actually proving something without really proving a damn thing.

The second problem I have with the "status-quo" argument is that it assumes that your opponent likes the status quo. Here's the problem with that: Education policies and practices are as wide and deep as a Las Vegas buffet. And there are some things people like and some things people do not. It is very possible to think that one reform practice is good and another is junk. However, in your simple mind, it is easier for you to ignore the buffet and lump it all together to have the American grunge match of "me vs them" on the big screen with your status-quo hot dog in your fat mouth. Trust me here: take the time and care to be specific about what you really mean instead of reducing everything down to some lame David and Goliath re-enactment that isn't even true.

Posted by: DHume1 | December 31, 2010 3:08 AM | Report abuse


I hope my fragile feelings will survive being called a moron by a random message board commentator. How can I go on?

You are correct that the country has been engaged in a non-stop series of "reform" efforts since a Nation at Risk. The vast majority of these effort have failed to meaningfully improve student achievement.

In my admittedly moronic mind, this means we need to pay very close attention to the rare instances in which academic achievement has substantially improved.

Call me crazy, but that's my story and I'm sticking to it.

Posted by: Ladner665 | December 31, 2010 2:36 PM | Report abuse

I know how you can go on: Get up in the morning and say to yourself, "It's time I go on a diet." Make it a New Year's resolution.

Posted by: DHume1 | December 31, 2010 7:45 PM | Report abuse

The comments to this entry are closed.

RSS Feed
Subscribe to The Post

© 2011 The Washington Post Company