Does KIPP shed too many low-performers?
My colleague Valerie Strauss, creator and proprietor of the fabulous The Answer Sheet blog on this Web site, recently encouraged a spirited debate over attrition rates at KIPP schools. I wrote my last book, "Work Hard. Be Nice" about the birth and growth of KIPP, the charter school network most successful in raising student achievement. (The official name is now just KIPP, not the Knowledge Is Power Program.)
I still follow KIPP closely. I want this blog to be the go-to place for anyone who wants to keep up with important developments in the network of 99 schools in 20 states and the District. Valerie has graciously agreed to allow me to put those recent KIPP posts from the debate here, so you can easily follow the lines of reasoning and can read my views.
It began with a great post (despite its polite digs at me) by Richard D. Kahlenberg, the Century Foundation senior fellow who has provided much original thinking on how to improve the education of disadvantaged children:
In the recent education debate between Valerie Strauss and Jay Mathews, a question arose about the attrition rates at the highly regarded Knowledge Is Power Program (KIPP) schools. The issue is important because if large numbers of weaker students drop out of KIPP’s rigorous program, it would be highly unfair to compare the test score gains won by the top KIPP students against the scores of all regular public school students – who include KIPP dropouts.
In the debate, Strauss mentioned some studies finding that KIPP schools “have had a very high attrition rate.” Mathews responded by saying it is a “myth that KIPP schools have poor retention rates” and cited a 2010 study that found that KIPP school “are doing about as well as regular schools in their neighborhoods” in terms of attrition.
Who’s right? While I respect Jay Mathews’s grasp of educational issues, on this question, the data overwhelmingly support Valerie Strauss’s skepticism.
In a rigorous 2008 study of five KIPP schools in the San Francisco Bay Area, researchers at SRI International found that an astounding 60% of KIPP students left over the course of middle-school. Moreover, the researchers found evidence that the 60% of students who did not persist through the tough KIPP regimen (a longer school day and week, and heavy doses of homework), tended to be the weaker students.
KIPP supporters, like Mathews, respond that a 2010 study of 22 KIPP schools by Mathematica found that the attrition rates were comparable to nearby high poverty public schools that also have lots of kids leave. Poor people tend to move frequently, so high attrition rates are to be expected at KIPP schools, it is argued.
The big difference between KIPP and regular public schools, however, is that whereas struggling students come and go at regular schools, at KIPP, student leave but very few new children enter. Having few new entering students is an enormous advantage not only because low-scoring transfer students are kept out but also because in the later grades, KIPP students are surrounded only by successful peers who are the most committed to the program.
(At this point Kahlenberg displays a graph showing the attrition rate at the KIPP Bay Area schools. You can find it on Valerie's blog.)
In the comments section of the Answer Sheet blog, when readers pointed out that KIPP schools don’t generally fill students back in, Mathews responded, “KIPP schools DO take in new students beyond the 5th grade.”
This is technically accurate, but as the figure above suggests, the vast majority of students enter during the 6th grade (a natural time to enter middle school) and then the total number of KIPP students in 7th and 8th grade falls precipitously.
The KIPP Bay-area schools cannot be dismissed as an outlier on the KIPP attrition question. Columbia University researcher Jeffrey Henig’s 2008 review of several studies found high attrition rates at a number of other KIPP schools.
It may well be, in fact, that high attrition rates are a key explanation for KIPP’s success in raising test scores. When KIPP tried to take over a regular public school – where the students are not self-selected, but are assigned to the school; and where students not only leave, but large number of students enter — KIPP abandoned the field after just two years. KIPP long ago realized that what we charge regular public schools with doing is far more difficult than what KIPP seeks to do.
There were many good comments, both con and pro, attached to Kahlenberg's piece. Here is one that caught my eye. (Ignore the misspelling of my last name. I am sure it was not intentional.
The difference between Valerie Strauss and Jay Matthews is the difference between a real reporter who actually digs for information and uses it objectively and a reporter with a book to sell. Posted by: adcteacher1 | January 4, 2011 8:02 AM
Still groggy from excessive football watching over the weekend, it took me a few days to catch up with the debate. Here is the comment I posted:
I have great respect for Rick and his work, but he is putting far too much weight on data from one KIPP region, and four-year-old data at that. (The Bay Area study came out in 2008, but it was reporting things that happened a couple of years before.) Henig says that some KIPP schools have high turnover rates, but that is not the same thing as saying they are throwing out kids. All the rest of the available evidence shows that KIPP schools on average lose no more kids than other local schools do, and in some cases, once the KIPP school is well established, they lose fewer students.
I think this discussion is fine, with many good points, but it irks me that commenters will suggest, based on one data point, that KIPP is kicking out low performing students. That is wrong. As one commenter pointed out, most of those students leave for the usual reasons, the parents move or they just prefer another school. One of the Bay Area schools lost many kids its first year because they recruited in one neighborhood, and then had to move across the city because they could not keep the facility they had in that neighborhood. Their turnover rate is less now.
As for the depth of reporting in my KIPP conclusions, Valerie is a terrific reporter, and I love her blog. But I think she will agree that on the subject of KIPP, my eight years of research for that book, including visits to more than 40 KIPP schools, has allowed me to learn more about that particular subject than she or Rick has, as good as they are. It is my specialty, and I appreciate the chance to defend what I know in exchanges like this. And don't forget, Rick's review of my book was a rave! Posted by: Jay Mathews | January 7, 2011 12:48 PM
Reading that comment now, I realize I was wrong to suggest that Kahlenberg was accusing KIPP of kicking out kids. (Although other commenters often do say that.) His point was more subtle. He was suggesting that attrition at KIPP, for whatever reason, was making its achievement gains look better than they actually were. I was saved from making more self-congratulatory, deceptive comments by a post from Jonathan Cowan, the KIPP Foundation's chief research and innovation officer, and Steve Mancini, KIPP's national spokesman, who addressed Kahlenberg's concerns:
On Jan. 3, The Answer Sheet featured a guest post by Richard Kahlenberg that highlighted attrition in KIPP schools.
We respect Mr. Kahlenberg’s right to question KIPP’s results, and we welcome healthy debate about the merits of KIPP’s philosophy and model. However, it is also important to clarify the fact base around the issues he raises.
At KIPP, we have a long standing commitment to transparency, continuous learning, and improvement. As such, we are always improving our data collection and reporting processes in order to share our successes and challenges. We focus on understanding the "health" of our schools: Are we serving the students who need us? Are our students staying with us? Are our students making academic progress? Are we fulfilling our promise to get kids to and through college? Are we creating a sustainable model?
Over the last few years, we have begun publicly reporting our performance as it relates to these questions. For instance, we publish our student mobility data in our annual Report Card, to illustrate whether our students are, in fact, staying with us. Our success depends on being held accountable for the results we produce for our kids.
In order to address specific criticisms raised in the piece, we ’d like to clarify Mr. Kahlenberg’s conflation of KIPP’s attrition statistics and our policies on “backfilling” empty student spots. In fact, these are two entirely separate issues, and should be addressed individually:
Assertion 1: KIPP’s success is due to high attrition and the fact that the “weakest” students leave.
Mr. Kahlenberg mentioned the June 2010 report by Mathematica Policy Research, but claimed that it does not tell the whole story when it comes to student mobility. In fact, the Mathematica report is very comprehensive, looking at 22 new and full-fledged schools over four years.
As Mr. Kahlenberg stated, the study found that attrition rates at KIPP schools nationwide were not systematically higher or lower than at comparable schools—some schools had higher attrition, some lower, some the same.
But the Mathematica report also had a second finding that Mr. Kahlenberg did not highlight: The vast majority of KIPP schools had a significant impact on achievement for all students who had ever attended, even if they didn’t complete all four years. Students who left the 22 schools during the study period were still counted in the report, which means the high achievement researchers found was not just a result of attrition. In fact, in conducting the analysis this way, Mathematica is holding KIPP accountable for all the students it ever enrolled, whether they stayed or left.
In his post, Mr. Kahlenberg relied on a study of KIPP Bay Area schools, published by SRI International in 2008, that found those schools to have unusually high levels of attrition. We absolutely agree that this study was rigorous and its findings are valid.
However, it was based on data from just five KIPP schools over a three-year period, and only one of those schools had reached full enrollment at the start of the study period. Thus, the SRI study does not account for how attrition rates at those schools have fallen as these KIPP schools have matured over the past four years.
Assertion 2: KIPP middle schools have high test scores because they do not enroll students after sixth grade.
Mr. Kahlenberg claimed that the reason Mathematica’s attrition results are flawed is because KIPP schools do not accept new students to make up for the ones they lose. He acknowledged that KIPP does take in new sixth-graders, but claimed that this is because sixth grade is “a natural time to start middle school.”
However, it is not the case that KIPP cuts off enrollment after sixth grade. Many KIPP middle schools, including those at KIPP DC, now regularly enroll new students at all grade levels, fifth through eighth. KIPP’s high schools also take students at all levels, from ninth to twelfth grade.
As more schools are reaching full enrollment and sustainability, this issue of “backfilling” classes is also subsiding.
The SRI study data Mr. Kahlenberg cites cuts off in 2006-07, when the eighth grade class was at 55% of the starting size of the entering fifth grade class. But that data is now several years old, and those numbers have improved dramatically. As of 2010-11, the KIPP Bay Area eighth grade class is at a full 86% of its starting fifth-grade size. We are working hard to increase that percentage even farther, at KIPP Bay Area and in all other regions.
As KIPP continues to grow, moving from start-up to sustainability, we have seen significantly reduced attrition rates and had success enrolling students at all grade levels. We remain focused on continuing to improve in these areas so we can set ever more students on the path to college and a better future.
Kahlenberg's response said in part:
Cowan and Mancini don’t dispute the data, but say that KIPP has gotten better at reducing attrition since the 2008 study was published, and that “many KIPP middle schools, including those at KIPP DC, now regularly enroll new students at all grade levels.” This is a very welcome development but could use some elaboration.
Cowan and Mancini say that KIPP is committed to transparency, and in that spirit, I’ll end with three questions:
1. How many students are now entering KIPP schools across the country during the seventh and eighth grades? Today, what is the aggregate difference between the size of the sixth grade KIPP classes two years ago and the eighth grade classes today? And how has the new influx of students in seventh and eighth grade affected KIPP test scores?
2. Which groups of students (by race, gender and income) are most likely to leave KIPP?
3. If KIPP wants to put the self-selection, attrition, and intake issues to rest, why doesn’t it simply start taking over regular public schools, educating the students who happen to live nearby, including those who move in during the course of middle school?
These are good questions, which I suspect the KIPP people will answer eventually. As a reporter for the last 45 years, I have a much experience trying to drag useful information out of large organizations, particularly about schools. KIPP ranks with the very best in responsiveness. If anyone has good questions about KIPP, or anything else I write about, that have not been adequately answered, just send them to me at firstname.lastname@example.org, and I will see what I can do to get the information.
Kahlenberg's last question is the most intriguing. KIPP has never tried hard to take over an existing school. The brief takeover he mentions, which occurred in Denver several years ago, was half-hearted and never had strong support among national KIPP leaders. It was a stop-gap situation with few good alternatives, and it was difficult to get the local cooperation necessary, KIPP people have told me. They said they never found a school leader who met their standards and was willing to make a long-term commitment to run the school.
When I raise the general question of taking over regular schools, KIPP people usually say it is hard enough to make their charter schools successful. I think they would agree with Kahlenberg that turning around a regular urban school is harder than creating a successful urban charter school, but would add that creating a successful urban charter school is still extremely difficult. The data prove them right. Nobody has done as well as KIPP in raising student achievement, and on average urban charter schools nationally are no better than the regular schools in their neighborhoods.
KIPP people also suggest that they would not attempt to fix a regular school unless they were allowed to use the tools they feel have been vital in making their charters successful. One KIPP leader once told me that KIPP could turn around a local school only if it was allowed to hire and fire faculty without regard to existing rules, something that was not going to happen.
KIPP would also insist, I suspect, on being allowed to institute its nine-hour school days, required summer school, homework requirements and discipline measures, and be free to change them when necessary. In other words, KIPP wouldn't try to take over a regular school unless allowed to run it as it runs its own schools -- with the only difference being that all neighborhood students would be able to attend. KIPP now takes all students who sign up for their schools and win the lottery if there is not enough space.
I wager some district superintendent, maybe a former KIPP teacher, will some day persuade a KIPP school leader to take a regular school under the conditions above, and see what happens. Rick Kahlenberg makes the good point that such an experiment would help KIPP, and the rest of us, learn what might really help our urban schools to change for the better.
| January 10, 2011; 6:16 PM ET
Categories: Jay on the Web | Tags: Jonathan Cowan, KIPP, KIPP attrition rates, KIPP taking over a regular school, Richard Kahlenberg, Steve Mancini
Save & Share: Previous: Learning inspired by teacher and textbook errors
Next: Obama on No Child Left Behind: Get me rewrite
Posted by: rasheeedj | January 11, 2011 7:13 PM | Report abuse
Posted by: jaymathews | January 11, 2011 9:10 PM | Report abuse
Posted by: JimmyKilpatrick | January 12, 2011 6:21 AM | Report abuse
Posted by: jbeeler | January 12, 2011 8:04 AM | Report abuse
Posted by: jbeeler | January 12, 2011 8:06 AM | Report abuse
Posted by: hainish | January 12, 2011 9:06 AM | Report abuse
Posted by: Nikki1231 | January 12, 2011 9:25 AM | Report abuse
Posted by: ericpollock | January 12, 2011 10:23 AM | Report abuse
Posted by: Jay Mathews | January 12, 2011 11:34 AM | Report abuse
Posted by: joshofstl1 | January 12, 2011 12:04 PM | Report abuse
Posted by: hainish | January 12, 2011 12:20 PM | Report abuse
Posted by: Jay Mathews | January 12, 2011 1:53 PM | Report abuse
Posted by: Nikki1231 | January 13, 2011 5:50 AM | Report abuse
Posted by: Cal_Lanier | January 13, 2011 1:26 PM | Report abuse