11488337_sIt has been quite a while since I wrote, critically, about Microsoft’s patch program, but the company’s latest patch woes have me a bit concerned.

Back in the day, when I first started reporting on computer security, Microsoft and its patch woes were a font of good story material that seemed never to empty. Back then (I’m talking 2002, here), Windows was the only security story worth talking about. And it seemed like some critical new vulnerability (or a worm that exploited it) was always in the offing. You could also count on Microsoft to act befuddled: downplaying the severity of the problem, sending mixed signals about when it would be fixed, or encountering some technical problems in the fixing of it. Mind you: this was great for us reporters. Every twist and turn in the weeks-long saga warranted its own, breathless coverage. After all, the stakes were pretty high. Remember MS.Blaster? Or how about SQL Slammer?

But, in short order, Microsoft got its stuff together. Bill Gates sent out his now famous Trustworthy Computing Memo in January 2002. After that, the changes came fast and furious. First, the company committed itself, as no other software maker has before or since, to using a Secure Development Lifecycle (SDL) that focused on rooting out exploitable vulnerabilities during the coding process.

Microsoft re-organized its development groups to create a designated team to handle patching and security incident response, The Microsoft Security Response Center (MSRC). And, ten years ago this month, the company pivoted from a patch culture of fires-stomping (responding to each security incident separately, and on its own timeline, adopting a disciplined, monthly patch release set for the second Tuesday of every month. That kind of thing is standard operating procedure these days. Back then, it was revolutionary.

Finally, Microsoft got wise to the issue of patch quality. After all, it doesn’t help your customers to receive a software patch in a timely manner, if their systems break when they apply said patch. In 2005, it introduced the Security Update Validation Program (SUVP), which allowed the company to test security updates in customer lab environments prior to release.

The cumulative effect of these changes was profound. As a reporter, covering Microsoft security patches became less like reporting from a crime scene and more like writing about the weather. Patch Tuesdays would come and Patch Tuesdays would go. Some were stormy - thick with critical updates, others not, but it wasn’t what you’d call “news.” It was just information.

Microsoft, itself, went from being defensive and cagey about its plans (which we reporters love) to maddeningly transparent. The company was always willing to put someone on the line to talk about that month’s patches. Even if they couldn’t give you all the details, they’d talk your ear off about the predictability and discipline of the SDL and SUVP process – killing you with kindness until your head dropped on the desk in boredom and defeat.

However, ten years after Microsoft announced its monthly patch program, there are worrying signs that the company may be losing some of the ground it gained over the last decade. Three times in the last six months, Microsoft has been forced to recall and then reissue software updates, following reports of problems from its customer base.

Most recently, last week, the company withdrew a botched patch for its Microsoft Office 2013 suite within hours of its release after customers complained that one component, KB 2871630, caused the folder list in Outlook to disappear.(Yikes!) In a move reminiscent of the days of yore, Microsoft downplayed the issue, using an unpublicized blog post on its (massive) Technet blog site to admit the error and suggest customers affected uninstall that item.

The September patch debacle follows a similar incident in August, when Microsoft also recalled two of its eight monthly patches, MS13-061 and MS13-066, after it discovered that the patches caused problems with the Microsoft Exchange Search Host Controller and Active Directory Federation Servers.

And, in April, Microsoft was forced to recall one of its monthly patches, MS13-036, which repaired vulnerabilities in a Windows Kernel-mode driver, after receiving reports that some customers were encountering the dreaded “blue screen of death” after installing the fix.

Problems with patches aren’t new. Microsoft has been forced to recall a patch before. But such incidents have been far and few between and the integrity of Microsoft patches has been above reproach.

That’s important. Patching vulnerable systems in a timely manner is one of the most important factors in thwarting would be hackers. And there are enough disincentives to patching without heaping worry on the backs of IT admins that their servers and desktops won’t be able to reboot after a patch is applied.

In each of the above cases, the patches were re-released without incident. But not before Microsoft’s customers voiced their displeasure: declaring that they are losing faith in Microsoft’s ability to deliver stable updates that are ready for production, or counting themselves lucky at having delayed deploying a patch.

“I can understand producing updates can be complex but you introduced these measures to improve quality, you're not following them and you're letting us down,” one customer using the name “Brian” commented in a Microsoft Exchange forum on Technet.

Trust gained and then lost can be difficult to restore. That means Microsoft will have to redouble efforts to ensure the integrity of its software updates, before customers begin looking elsewhere.

About Paul Roberts

Paul Roberts is an experienced technology writer and editor that has spent the last decade covering hacking, cyber threats, and information technology security, including senior positions as a writer, editor and industry analyst. His work has appeared on NPR’s Marketplace Tech Report, The Boston Globe, Salon.com, Fortune Small Business, as well as ZDNet, Computerworld, InfoWorld, eWeek, CIO , CSO and ITWorld.com. He was, yes, a guest on The Oprah Show — but that’s a long story. You can follow Paul on Twitter here or visit his website The Security Ledger.

Comments (2)

Richard Steven Hack | September 20, 2013 11:51 pm

On XP, their entire patch mechanism was a disaster. I don't know how many times I've had to rebuild the patch subsystem for clients or find a way to fix a patch that wouldn't install.

The .NET subsystem was the worst, with the Visual software right behind - hardly anything ever updated properly - and the fix entailed ripping out all of .NET and reinstalling it.

Windows 7 seems to be much better, fortunately. I haven't seen too much failure of the update mechanism - none, actually.

However... "the company committed itself, as no other software maker has before or since, to using a Secure Development Lifecycle (SDL) that focused on rooting out exploitable vulnerabilities during the coding process."

Color me unconvinced. Based on the reports from internal Microsoft personnel during the Vista development debacle, I suspect they could do a lot more.

Not that open source is much better. It will be another decade or two before the industry really commits to reliable and secure software - and that will only happen after the corporate and end user world is beaten against the wall with ubiquitous security and reliability issues. Might not happen until those issues start causing people to die regularly...

Matt | September 23, 2013 7:06 pm

It's no surprise that Microsoft's security division, it disappointing people. They always have and probably always will.

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.