It has been quite a while since I wrote, critically, about Microsoft’s patch program, but the company’s latest patch woes have me a bit concerned.
Back in the day, when I first started reporting on computer security, Microsoft and its patch woes were a font of good story material that seemed never to empty. Back then (I’m talking 2002, here), Windows was the only security story worth talking about. And it seemed like some critical new vulnerability (or a worm that exploited it) was always in the offing. You could also count on Microsoft to act befuddled: downplaying the severity of the problem, sending mixed signals about when it would be fixed, or encountering some technical problems in the fixing of it. Mind you: this was great for us reporters. Every twist and turn in the weeks-long saga warranted its own, breathless coverage. After all, the stakes were pretty high. Remember MS.Blaster? Or how about SQL Slammer?
But, in short order, Microsoft got its stuff together. Bill Gates sent out his now famous Trustworthy Computing Memo in January 2002. After that, the changes came fast and furious. First, the company committed itself, as no other software maker has before or since, to using a Secure Development Lifecycle (SDL) that focused on rooting out exploitable vulnerabilities during the coding process.
Microsoft re-organized its development groups to create a designated team to handle patching and security incident response, The Microsoft Security Response Center (MSRC). And, ten years ago this month, the company pivoted from a patch culture of fires-stomping (responding to each security incident separately, and on its own timeline, adopting a disciplined, monthly patch release set for the second Tuesday of every month. That kind of thing is standard operating procedure these days. Back then, it was revolutionary.
Finally, Microsoft got wise to the issue of patch quality. After all, it doesn’t help your customers to receive a software patch in a timely manner, if their systems break when they apply said patch. In 2005, it introduced the Security Update Validation Program (SUVP), which allowed the company to test security updates in customer lab environments prior to release.
The cumulative effect of these changes was profound. As a reporter, covering Microsoft security patches became less like reporting from a crime scene and more like writing about the weather. Patch Tuesdays would come and Patch Tuesdays would go. Some were stormy - thick with critical updates, others not, but it wasn’t what you’d call “news.” It was just information.
Microsoft, itself, went from being defensive and cagey about its plans (which we reporters love) to maddeningly transparent. The company was always willing to put someone on the line to talk about that month’s patches. Even if they couldn’t give you all the details, they’d talk your ear off about the predictability and discipline of the SDL and SUVP process – killing you with kindness until your head dropped on the desk in boredom and defeat.
However, ten years after Microsoft announced its monthly patch program, there are worrying signs that the company may be losing some of the ground it gained over the last decade. Three times in the last six months, Microsoft has been forced to recall and then reissue software updates, following reports of problems from its customer base.
Most recently, last week, the company withdrew a botched patch for its Microsoft Office 2013 suite within hours of its release after customers complained that one component, KB 2871630, caused the folder list in Outlook to disappear.(Yikes!) In a move reminiscent of the days of yore, Microsoft downplayed the issue, using an unpublicized blog post on its (massive) Technet blog site to admit the error and suggest customers affected uninstall that item.
The September patch debacle follows a similar incident in August, when Microsoft also recalled two of its eight monthly patches, MS13-061 and MS13-066, after it discovered that the patches caused problems with the Microsoft Exchange Search Host Controller and Active Directory Federation Servers.
And, in April, Microsoft was forced to recall one of its monthly patches, MS13-036, which repaired vulnerabilities in a Windows Kernel-mode driver, after receiving reports that some customers were encountering the dreaded “blue screen of death” after installing the fix.
Problems with patches aren’t new. Microsoft has been forced to recall a patch before. But such incidents have been far and few between and the integrity of Microsoft patches has been above reproach.
That’s important. Patching vulnerable systems in a timely manner is one of the most important factors in thwarting would be hackers. And there are enough disincentives to patching without heaping worry on the backs of IT admins that their servers and desktops won’t be able to reboot after a patch is applied.
In each of the above cases, the patches were re-released without incident. But not before Microsoft’s customers voiced their displeasure: declaring that they are losing faith in Microsoft’s ability to deliver stable updates that are ready for production, or counting themselves lucky at having delayed deploying a patch.
“I can understand producing updates can be complex but you introduced these measures to improve quality, you're not following them and you're letting us down,” one customer using the name “Brian” commented in a Microsoft Exchange forum on Technet.
Trust gained and then lost can be difficult to restore. That means Microsoft will have to redouble efforts to ensure the integrity of its software updates, before customers begin looking elsewhere.