Counting Civilian Deaths in Iraq
"Civilian deaths of all categories, less natural causes, have also declined considerably, by over 45 percent Iraq-wide, since the height of the sectarian violence in December."
--General David Petraeus, congressional testimony, September 11, 2007.
"The reports that you provide to us really require the willing suspension of disbelief. ... If you look at all the evidence that's been presented, overall civilian deaths have risen."
--Sen. Hillary Clinton, responding to Petraeus during his Senate testimony.
Which of these two statements is correct? The tally of violence-related civilian deaths in Iraq has become one of the most contentious issues in the political debate over the war. Our goal here is to take a close look at the factual foundations of the various claims and counter-claims. For the last two weeks, we have engaged in lengthy e-mail exchanges with Petraeus's spokesman in Baghdad, obtained data supplied by the Multi-National Force-Iraq (MNF-I) and conducted interviews with data-crunching experts in the United States and Britain, seeking a clearer picture of how military officials arrived at the "over 45 percent" claim made by Petraeus during his testimony to Congress. Among our preliminary findings:
* The Petraeus claim rests on combining two distinct data sets, one more reliable than the other, according to MNF-I. The most authoritative data set comes from direct reports from Coalition units dispatched to the scene of major incidents, such as car bombs and small arms fire. Since this reporting is invariably incomplete (minor incidents such as executions and random killings are usually not investigated by Coalition units), MNF-I tops up the data with "host nation" statistics, such as morgue and police reports.
* The data collected directly by Coalition units (see graph below) show a much more modest reduction in Iraqi civilian deaths since last December than the data presented to Congress by Petraeus.
* Petraeus's spokesman, Colonel Steven A. Boylan, acknowledges the "host nation" data used by MNF-I statisticians is frequently "unverified" and "possibly inflated." In an email, he raised a particular red flag about the "host nation" data for December 2006, the base line for Petraeus's claim of an "over 45-percent drop" in Iraqi civilian deaths. He said that the December 2006 data contained "a large number of unverified host nation reports of dead civilians."
* An analysis of the Iraqi civilian death toll by the unofficial London-based group, Iraq Body Count shows a much smaller decline over the period cited by Petraeus. According to Iraq Body Count's latest figures, the civilian war-related death toll declined from around 2,600 to around 2,200 over this period, a drop of roughly fifteen percent.
* There is little evidence to support claims of an actual increase in civilian deaths over the last eight or nine months, other than on an individual month-by-month basis.
Over the last month, a lively debate has erupted in both the media and the think tank community over the statistical methodology employed by the U.S. military in calculating civilian body counts.
In a September 6 article in The Washington Post, my colleague Karen DeYoung revealed deep divisions within the U.S. government over how to interpret violence statistics. She followed up on September 25 with a detailed look at how the U.S. military defines "sectarian killings," a
subset of the overall civilian casualty statistics. The coverage prompted fierce debates, with liberal think tanks, such as the Center for American Progress, weighing in with their views, and the conservative Heritage Foundation posting its analysis here. Most recently, the Council on Foreign Relations has published a statistical analysis of the Petraeus data, and an online debate featuring two experts, one of whom sides broadly with Petraeus (with some reservations) and the other who argues flatly that "violence in Iraq is not going down."
The following graph, provided by MNF-I and published here for the first time, helps us deconstruct the civilian death statistics Petraeus delivered in his testimony to Congress. (A larger-size version of the graph is here.)
To understand the significance of this graph, it is necessary to explain the meaning of each line.
* The blue line in the center represents the Iraqi civilian death data used by Petraeus in his September 10-11 testimony to Congress. It corresponds to Slide Three in his presentation and provided the basis for his "over 45 percent drop" claim. It includes data verified by Coalition forces plus verified and "unverified" data supplied by the Iraqi government.
* The orange line at the top represents civilian casualties, including both dead and injured, reported directly by Coalition forces. A version of these figures appeared in an "Average Daily Casualties in Iraq" chart submitted to Congress this month by the Department of Defense. See the full report here.
* The yellow line at the bottom represents Iraqi civilian death data collected and verified by coalition forces. As far as we are aware, it has never been published before.
As a guide to the true situation in Iraq, each line has its pluses and minuses. As you might expect, the orange and yellow lines show similar trends, although the orange casualty line shows greater movement than the yellow death line. But they are also incomplete, as they omit most minor incidents, such as individual murders and executions. By including unverified, "possibly inflated" data, the blue line shows a much steeper decline in the death toll over the period Petraeus highlighted than the yellow line below.
The mix and match procedures used by MNF-I to combine the coalition-verified deaths and "unverified" host nation reports require further investigation. The methods used for adjusting the Coalition-reported figures are unclear. An analysis of a chart provided by MNF-I shows that military statisticians added some 1,750 "host nation reported deaths" to the coalition-verified baseline in December 2006. In August 2006, by contrast, at the end of Petraeus's reporting period to Congress, only 500 "host nation reported" deaths were included. Over the same time-frame, Coalition-reported deaths fell only modestly, from around 1,250 to around 1,050.
Boylan defended the use of the blue line by Petraeus in his testimony to Congress by saying that it was "as close as we can get to total truth on dead civilians...We used this data to ensure that there can be no claims as to cherry-picking or deflating data by only using Coalition reporting/data as we incorporated host nation data into the testimony. Petraeus purposefully chose to use the higher (and possibly inflated) blue line data to avoid any charge that he was trying to make everything look good or more positive. This is as realistic as it comes."
Boylan declined several requests to explain the dramatic 70 percent drop in "host-nation reported" deaths between December 2006 and August 2007. He said it was difficult to "get to the level [of detail] you wish" without a personal visit by the Fact Checker to Baghdad. He invited the Fact Checker to come to Baghdad "for a week" to sit down with military number-crunchers. The Fact Checker replied that he is ready to accept the invitation but needs the approval of his editors.
For the sake of fairness, we are posting Boylan's initial September 22 statement in full here.
A detailed unofficial examination of the death toll among Iraqi civilians is offered by Iraq Body Count, an independent, London-based organization that bases its analysis on multiple media reports. The IBC data has formed the basis for the tracking of civilian deaths in Iraq by the Brookings Institution in its widely consulted Iraq Index. An analysis of the last eighteen months of Iraqi death toll figures compiled for the Fact Checker by IBC diverges at times from the "blue line" data used by Petraeus. Like the "orange line" data, it suggests a much smaller drop in Iraqi casualties since a peak in October-November 2006 than the "blue line" data.
Here is what John Sloboda, executive director of Iraq Body Count, has to say about the Petraeus testimony to Congress:
"It is interesting that there is general agreement between Iraq Body Count and Petraeus/MNF-I up until December 2006. Our figures are usually higher than theirs -- we may be including some things they are perhaps not, like the deaths of some Iraqi policemen -- but our peaks and troughs coincide with Petraeus/MNF-I very clearly for most of 2006. Then, quite abruptly, in December 2006, the Petraeus numbers rise above ours for the first time. But as soon as the surge begins, they plunge way below our level.
"Our methods have been consistent throughout. Why the sudden discrepancy? Is the reason for it purely technical? It's hard to tell from the graphs alone. We can provide documentary backup for every one of our figures, day by day, incident by incident. It is impossible to fully assess the Petraeus figures unless the data which lies behind it is revealed, on an incident-by-incident basis. Only then could one compare the two sets of data and nail down the source and nature of this discrepancy.
"It is, of course, welcome that the Pentagon is finally taking account of Iraqi civilian deaths and injuries in a methodical manner. However, unless this is done as transparently as possible, doubts over their validity may be all that these graphs engender."
The analysis by the Council on Foreign Relations claims that Iraq Body Count data show a steeper decline in Iraqi civilian deaths during the first eight months of 2007 than data used by MNF-I. The author of the Council's report, Stephen Biddle, said in an interview, however, that he had relied on old Iraq Body Count data downloaded from the Internet rather the most recent data endorsed by the organization. Iraq Body Count agrees that there has been some decline in civilian deaths since December, but says that it is less pronounced than claimed by MNF-I.
Data collected by the Washington Post Baghdad bureau from the Iraqi Health Ministry show an even steeper rate of decline in the civilian death toll than that cited by Petraeus. The Post data reflect bodies delivered to morgues, excluding bodies buried directly by families. It is noteworthy that the Post data show a drop in civilian deaths between July and August 2007, a month that witnessed massively destructive suicide attacks against Yazidi villages in northern Iraq. (According to the international humanitarian organization Red Crescent, the Yazidi attacks claimed at least five hundred lives.) The Baghdad bureaus of several other prominent news organizations, including the New York Times, the Associated Press, and Reuters reported an increase in Iraqi civilian deaths over the same two months. One possible explanation for the disrepancy is that many Yazidi families buried their dead directly, rather than sending them to morgues, but this too requires further investigation.
SOURCES: Unofficial data from Iraqi Health Ministry | The Washington Post - September 5, 2007
Questions can also be raised about Clinton's claim that "overall, civilian deaths have risen" in Iraq. She was imprecise about exactly what period she was talking about. She was reading almost verbatim from a Center for American Progress report that addressed the general situation in Iraq since the beginning of the new surge strategy in January 2007. Asked to provide factual support for the senator's remarks, the Clinton campaign cited this September 2 article in the New York Times. But the article talks only about a rise in civilian deaths Iraq-wide between July and August this year, accompanied by a fall in civilian deaths in Baghdad. A Clinton spokesman, Philippe Reines, now says the senator was speaking about "the month prior to the delivery of the [Petraeus] report to Congress."
We give the last word (for now) to Colonel Boylan:
"We completely stand by our data as the most robust and accurate data available. We have no reason to falsely report any of this and constantly strive to improve both collection and analysis."
(UPDATED Oct 1, 1:22 p.m. ET: After this item was posted, Boylan contacted the Fact Checker to say that he sent an e-mail message on September 27 (last Thursday) offering a phone interview with a statistical expert from MNF-I, in addition to written statements already provided. The Fact Checker never received this e-mail. He first requested such a briefing on September 21. MNF-I now says it will make the chief of its statistical assessments office available for an interview on Tuesday. We will keep you informed.)
The Pinocchio Test
This issue is too complex for a snap Pinocchio rating. In this case, our goal is to shed light on a complicated statistical debate, rather than slap Pinocchio or Geppetto labels on either Petraeus or Clinton. There will be plenty of opportunity on this site to debate and question the Fact Checker's methodology, including a planned "Ask the Fact Checker" open forum.
MNF-I's readiness to collect Iraqi civilian death data and answer questions about its methodology contrasts with the position taken by former Centcom commander Gen. Tommy Franks who declared during the Afghan war, "we don't do body counts." (News conference, Bagram Air Base, March 2002.) The American military appears to have definitively abandoned the "Tommy Franks doctrine" of civilian casualty statistics.
But without more information from MNF-I on the methods used to combine American and Iraqi data, it is impossible to tell whether Petraeus is "cooking the books," in the charged language of the recent MoveOn.org ad. The jury is still out, but the debate is well underway.
* Congressional Research Service report on Iraq civilian deaths
* Council on Foreign Relations statistical analysis by Stephen Biddle
* Council on Foreign Relations debate on whether violence is going down in Iraq
* White House Iraq Fact Check: Responding to Key Myths
* September 2007 report on Iraqi benchmarks by the General Accountability Office.
* Analysis by former U.S. statistician in Baghdad (posted on Heritage Foundation Web site)
| October 1, 2007; 8:01 AM ET
Categories: Gov Watch, Iraq, Verdict Pending
Save & Share: Previous: Newspaper Cures Skin Rash
Next: Young Black Males Headed for Extinction?
Posted by: Urbi | October 1, 2007 9:36 AM | Report abuse
Posted by: John Tirman | October 1, 2007 9:59 AM | Report abuse
Posted by: dortuna | October 1, 2007 10:30 AM | Report abuse
Posted by: barb | October 1, 2007 10:43 AM | Report abuse
Posted by: Greg Sanders | October 1, 2007 11:25 AM | Report abuse
Posted by: Joe | October 1, 2007 11:53 AM | Report abuse
Posted by: John | October 1, 2007 12:51 PM | Report abuse
Posted by: Judge C. Crater | October 1, 2007 1:38 PM | Report abuse
Posted by: Craig Jones | October 1, 2007 1:59 PM | Report abuse
Posted by: Crispin Pierce | October 1, 2007 2:20 PM | Report abuse
Posted by: trudyj | October 1, 2007 2:29 PM | Report abuse
Posted by: Craig Jones | October 1, 2007 3:07 PM | Report abuse
Posted by: frankie d | October 1, 2007 3:19 PM | Report abuse
Posted by: RetCombatVet | October 1, 2007 3:45 PM | Report abuse
Posted by: Mark | October 1, 2007 4:04 PM | Report abuse
Posted by: emma | October 1, 2007 4:05 PM | Report abuse
Posted by: frankie d | October 1, 2007 4:20 PM | Report abuse
Posted by: John | October 1, 2007 5:41 PM | Report abuse
Posted by: Headin da Sand | October 1, 2007 7:46 PM | Report abuse
Posted by: trudyj | October 1, 2007 8:31 PM | Report abuse
Posted by: CriticalThought | October 1, 2007 9:49 PM | Report abuse
Posted by: joseph kenny | October 1, 2007 9:56 PM | Report abuse
Posted by: david | October 2, 2007 12:37 AM | Report abuse
Posted by: david | October 2, 2007 12:38 AM | Report abuse
Posted by: Paulie200 | October 2, 2007 3:17 AM | Report abuse
Posted by: Anonymous | October 2, 2007 8:41 AM | Report abuse
Posted by: Stephen Biddle | October 2, 2007 11:47 AM | Report abuse
Posted by: tom | October 2, 2007 2:18 PM | Report abuse
Posted by: Engram | October 3, 2007 12:24 PM | Report abuse
Posted by: trudyj | October 3, 2007 3:19 PM | Report abuse
Posted by: Anonymous | October 8, 2007 1:07 PM | Report abuse
Posted by: jerry rubin | October 10, 2007 6:42 PM | Report abuse
Posted by: Major Herrera | October 16, 2007 9:46 PM | Report abuse
Posted by: LaAngeloMysterioso | October 17, 2007 5:01 AM | Report abuse
Posted by: Charisse Martin | October 17, 2007 3:28 PM | Report abuse
Posted by: bilbosimca | October 18, 2007 12:04 PM | Report abuse
Posted by: twerner | October 19, 2007 1:06 PM | Report abuse
Posted by: joop van de Swaluw | October 19, 2007 1:13 PM | Report abuse
Posted by: T Ruth | October 20, 2007 3:15 AM | Report abuse
Posted by: heather | November 2, 2007 11:19 AM | Report abuse
Posted by: Tracey Walsh | November 4, 2007 10:47 AM | Report abuse
Posted by: ragqlqnsuk | November 23, 2007 5:54 AM | Report abuse
Posted by: xkbsjtqcwb | November 25, 2007 11:34 AM | Report abuse
Posted by: espmljfy kcvhria | January 17, 2008 5:07 AM | Report abuse
Posted by: ptxraco orbpyh | January 17, 2008 5:08 AM | Report abuse
The comments to this entry are closed.