A Time to Patch
A few months back while researching a Microsoft patch from way back in 2003, I began to wonder whether anyone had ever conducted a longitudinal study of Redmond's patch process to see whether the company was indeed getting more nimble at fixing security problems.
For many years, Microsoft has been criticized for taking too long to issue patches, especially when compared with patch releases for flaws found in operating systems or software applications maintained by the open source community, such as Linux or Mozilla's Firefox browser. But I wanted to find out for myself just how long Microsoft takes on average to issue fixes for known software flaws.
Finding no such comprehensive research, Security Fix set about digging through the publicly available data for each patch that Microsoft issued over the past three years that earned a "critical" rating. Microsoft considers a patch "critical" if it fixes a security hole that attackers could use to break into and take control over vulnerable Windows computers.
For each patch, Security Fix looked at the date Microsoft Corp. was notified about a problem and then how long it took the company to issue a fix for said problem. In most cases, information about who discovered the vulnerability and when they reported it to Microsoft or disclosed it in public was readily available through various citations by Mitre, which maintains much of that data on the common vulnerabilities and exposures (CVE) list.
In some cases, however, that submission or disclosure date was not publicly available, and required Security Fix to contact the individual discoverer and get the dates directly from them. In about a dozen cases, the discoverer of a vulnerability did not respond to information requests or the flaw appeared to have been found internally at Redmond, and in those instances Microsoft filled in the blanks.
Here's what we found: Over the past three years, Microsoft has actually taken longer to issue critical fixes when researchers waited to disclose their research until after the company issued a patch. In 2003, Microsoft took an average of three months to issue patches for problems reported to them. In 2004, that time frame shot up to 134.5 days, a number that remained virtually unchanged in 2005.
Below are three spreadsheets detailing our findings for the past three years. The documents are downloadable either as Microsoft Excel files or regular HTML files:
In the first column of each spreadsheet, you should see a hyperlinked MS number that will take you to the Microsoft advisory for that patch. Next to that column is a link to the CVE entry, which contains quite a bit more information about how each flaw was discovered and by whom.
The data show that one area where Microsoft appears to be fixing problems more quickly is when the company learns of security holes in its products at the same time as everyone else. Advocates of this controversial "full disclosure" approach believe companies tend to fix security flaws more quickly when their dirty laundry is aired for all the world to see, and at least on the surface that appears to be the case with Microsoft.
It is important to note, however, that in nearly all full-disclosure cases cited here, news of the vulnerability was also issued alongside computer code demonstrating how attackers might exploit the flaw.
In cases where Microsoft learned of a flaw in its products through full disclosure, the company has indeed gotten speedier. In 2003, it took an average of 71 days to release a fix for one of these flaws. In 2004 that time frame decreased to 55 days, and in 2005 shrank further to 46 days.
The company also seems to have done a better job convincing security researchers to give it time to develop a patch before going public with their vulnerability findings. In 2003, Microsoft learned of at least eight critical Windows vulnerabilities through full disclosure. Last year, this happened half as many times.
I spoke at length about this project with Stephen Toulouse, a security program manager at Microsoft. (Toulouse's team also verified the data in the Excel spreadsheets that accompany this post). Toulouse said that if Microsoft is taking longer to release patches for known vulnerabilities, it is because the company has placed a renewed focus on ensuring that each patch comprehensively fixes the problem throughout the Windows operating system and that each fix does not introduce new glitches in the process.
Toulouse said developing a patch to mend a security hole is usually the easiest part. Things get more problematic, he said, during the testing process. If testers find a bug, the patch developers incorporate the fix into all relevant portions of the patch and the testing process is reset, forcing the testers to start from scratch. What's more, Microsoft also has been more willing of late to give the discoverers of each vulnerability the opportunity to vet the patches before they are released to the public.
Toulouse pointed to one particularly problematic patch that took the company 200 days to fix: a vulnerability in a component of Windows (and many other networking applications) known as ASN.1, at the time considered the largest vulnerability in the history of the Windows operating system. In the course of testing the patch for that flaw -- reported by security researchers at Aliso Viejo, Calif.-based eEye Digital Security -- Microsoft was forced to reset the process at least twice as internal developers found additional problems that were being masked by previously unknown glitches in the fix.
"We learned that it's far better for us to find those issues than for customers to run into them," Toulouse said.
Some of those lessons Microsoft learned when it tried to fix a critical flaw in Windows that was later exploited by the infamous Blaster worm. Microsoft turned around patch for that vulnerability, reported by researchers in the hacker group The Last Stage of Delirium, in just 38 days because it recognized that while the initial fix for the problem might not have not eradicated the flaw, there was a great deal of concern within Microsoft "about the breadth and depth of the vulnerability."
Two days after Microsoft released the patch, researchers alerted Microsoft that the flaw was present in three other areas of the operating system that the initial fix did not address. Roughly two weeks after that, the Blaster worm would infect millions of Windows PCs worldwide. Some security experts believe the worm may have been developed with the help of the initial Microsoft patch, which could have given the worm's authors a better idea of how to exploit the flaw.
"It was a conscious decision at the time to release that patch so quickly, but we later looked back and decided we really should have conducted a more thorough review process," Toulouse said.
According to Toulouse, Blaster resulted in two key changes at Microsoft. For starters, the company instituted a more thorough patch-review process across all company product teams that had a hand in developing the original vulnerable code. Microsoft also "retasked" its Secure Windows Initiative Team to research and attack each vulnerability the way a malicious hacker might.
"That team's job is to take the vulnerability, turn it sideways and upside down and to think 'Is there any other way to exploit this?'" Toulouse said.
I shared some of this data with a few of the security researchers and organizations credited with discovering flaws in the above-mentioned advisories, and got mixed responses to Microsoft's claims.
Pete Allor, manager of the X-Force vulnerability research division at Atlanta-based Internet Security Systems, praised Microsoft for "doing a fantastic job over the past year and a half on the [quality assurance] side of patching. We're not seeing the recalls and reissues that we used to. What we're hearing in today's corporate environment is, 'Make sure you get it right the first time. We don't want to hear how a patch is broken because you didn't 'take the time."
Not everyone sees Microsoft's recent patch efforts in such glowing light. Marc Maiffret, "chief hacking officer" for the aforementioned eEye, noted that the longer a patch is in the works, the longer customers remain unprotected. Maiffret says it is not uncommon for exploits to be available in the malicious hacker underground for vulnerabilities that have been reported in the meantime by well-meaning security researchers.
"You'd think that by taking that much longer on patches Microsoft is being more thorough, but that's not always the case as we've seen," Maiffret said. "The truth is that unpatched Windows flaws have a value to the underground community, and it is not at all uncommon to see these things sold or traded among certain groups who use them by quietly attacking just a few key targets. So, the longer Microsoft takes to patch vulnerabilities the longer they are leaving customers exposed."
Last Thursday, Microsoft released a patch to fix a critical flaw in the way Windows renders certain image files. That update, which mended a 0day ("zero day") vulnerability for which an exploit was publicly disclosed and very soon in use by attackers, took Microsoft just 10 days to produce, though the company was able to take some pointers from an unofficial patch that was released by an independent security researcher. Because the patch was issued in 2006, however, Security Fix did not include those 10 days in the 2005 time-to-patch averages.
I mention the WMF patch because earlier this week security researchers posted to the public Bugtraq software vulnerability list some exploit code for at least two more security flaws in the same WMF engine Microsoft patched last week. While those flaws (at least for now) are considered less dangerous than the problem Redmond fixed last week, it does raise questions about the Microsoft team charged with finding these problems. The vulnerabilities have apparently been present in the Windows operating system code dating back to Windows 3.0. Toulouse maintains that Microsoft had already flagged those glitches prior to the exploit code posting on Bugtraq, but because the company didn't see them as a big security threat, it did not hold up the WMF patch to include fixes for them.
One final note: Security Fix did not attempt to determine whether there was a correlation between the speed with which Microsoft issues patches and the quality or effectiveness of those updates. A real glutton for punishment might be able to learn just how many Windows patches were later updated with subsequent fixes -- either because the initial patch failed to fully fix the problem or introduced new troubles. I purposely did not undertake that task, in part because I figured I'd still be working on the project this time next year if I did.
I'd like to thank everyone who helped me assemble the data in the above graphs -- including (but certainly not limited to): Cesar Cerrudo, "Fozzy," Joao Gouveia, Maolin Gu, Kostya Kortchinsky, Marc Maiffret, Brett Moore, and Peter Winter-Smith. Please forgive me if I have forgotten to name anyone, and if I did just send me an e-mail and I'll update this post.
Posted by: H. Carvey | January 11, 2006 9:39 AM | Report abuse
Posted by: slightly confused | January 11, 2006 11:57 AM | Report abuse
Posted by: Bk | January 11, 2006 12:02 PM | Report abuse
Posted by: Adam | January 11, 2006 1:29 PM | Report abuse
Posted by: Eric Hanson | January 11, 2006 1:50 PM | Report abuse
Posted by: John | January 11, 2006 2:40 PM | Report abuse
Posted by: Mac Happy | January 11, 2006 3:15 PM | Report abuse
Posted by: J Arthur Rank | January 11, 2006 3:32 PM | Report abuse
Posted by: Arthur C. Korn | January 11, 2006 3:44 PM | Report abuse
Posted by: N. Meyer | January 11, 2006 4:27 PM | Report abuse
Posted by: GTexas | January 11, 2006 4:32 PM | Report abuse
Posted by: Tom | January 11, 2006 4:41 PM | Report abuse
Posted by: ajldfaldfaljfd` | January 11, 2006 5:22 PM | Report abuse
Posted by: Bill Myers | January 11, 2006 5:47 PM | Report abuse
Posted by: Bill Myers | January 11, 2006 5:50 PM | Report abuse
Posted by: Indy | January 11, 2006 6:34 PM | Report abuse
Posted by: B. Ferjulian | January 11, 2006 6:34 PM | Report abuse
Posted by: Bk | January 11, 2006 7:26 PM | Report abuse
Posted by: klone | January 11, 2006 8:07 PM | Report abuse
Posted by: Wayne | January 11, 2006 8:25 PM | Report abuse
Posted by: Mike B. | January 11, 2006 8:46 PM | Report abuse
Posted by: Tom K | January 11, 2006 9:02 PM | Report abuse
Posted by: Rick Molera | January 11, 2006 9:30 PM | Report abuse
Posted by: Mac Happy | January 12, 2006 7:40 AM | Report abuse
Posted by: Arlene Montemarano | January 12, 2006 8:08 AM | Report abuse
Posted by: MM | January 12, 2006 9:13 AM | Report abuse
Posted by: keydet89 | January 13, 2006 7:12 AM | Report abuse
Posted by: Dominic White | January 13, 2006 8:32 AM | Report abuse
Posted by: Sergio Hernando | January 13, 2006 9:13 AM | Report abuse
Posted by: Justin | January 13, 2006 7:02 PM | Report abuse
Posted by: Cornelio Hopmann | January 20, 2006 10:19 AM | Report abuse
The comments to this entry are closed.