Skip to main content
August 25, 2014

Secure Development - One Bathroom Break At A Time

Google went to great lengths to educate their developers about the benefits of security testing - even developing educational materials specifically to be read on the toilet.

There's enough evidence in favor of the use of security testing throughout the development cycle as to make "debates" about it moot. Still, many software development operations still lack a comprehensive and consistent approach to testing. Why? One of the most commonly cited reasons is the development "culture." That's a fuzzy term that encompasses a lot of things. Often it boils down to personal resistance on the part of (influential) developers and managers within an organization, or adherence to wrong - headed tradition. "We don't do that here." Full stop. It goes without saying that, if you want to change development practices, you need to change the development culture within your organization. But how?
Mike Bland

Mike Bland


Back in June, Mike Bland, a former Google Engineer, penned a great blog post that provides something of a roadmap to implementing a culture of secure development and testing. Bland worked at Google from 2005 to 2011 and, for much of time, was Google's "Testing Evangelist," helping to implement a system for thorough application testing and, more importantly, making practices like unit testing for developed code a cultural norm at Google. Bland's post is worth printing out and reading. He starts off with an analysis of the Apple GoTo Fail and Heartbleed vulnerabilities, and how they're object lessons for the importance of unit testing. (Veracode tackled some of the same issues regarding Goto Fail in this blog post.) But Bland also wraps in a (substantial) guide to developing a culture of secure development and testing. Bland weighs in on the relative importance and merits of various types of tests - integration testing vs. unit testing vs. fuzzing. And he talks about the organizational challenges of growing a culture of testing at Google. Bland says that, contrary to what you may believe, Google's engineering-heavy culture was not hospitable soil for the adoption of a culture of unit testing. Despite the company's considerable resources, Bland notes that "vast pools of resources and talent" within the company often got in the way by "reinforc(ing) the notion that everything is going as well as possible, allowing problems to fester in the long shadows cast by towering success." "Google was not able to change its development culture by virtue of being Google," Bland writes. "Rather, changing Google's development culture is what helped its development environment and software products continue to scale and to live up to expectations despite the ever-growing ranks of developers and users." Bland describes an all-hands approach to bending Google's engineering practice in the direction of more testing. Resistance, he said, was the byproduct of complex forces: a lack of proper developer education in unit testing practices and "old tools" that strained to scale with the pace of Google's development. Bland's response was to form what he calls a "Testing Grouplet" within Google that served as a support group and community for like-minded folks who wanted to implement unit testing procedures. That group operated like testing guerillas, developing and driving the adoption of new tools that made unit testing less painful, sharing best practices and engaging in testing propaganda within the development organization. One of the most successful initiatives was "Testing on the Toilet," a circa 2006 program in which Grouplet members designed short (one page) lessons on a variety of topics ("Don't Put Logic in Tests," "Debug IDs") then plastered hundreds of Google bathroom stalls worldwide with the *ahem* tactical reading material. The idea here isn't to shock folks. Rather: its to take the role of education in your secure development program seriously. Bland said the bathroom literature was part of a multi-pronged education effort that also included more common elements like brown bag lunches and a guest speaker series. Check out Mike's full post on building a unit testing culture at Google here.

Paul Roberts is an experienced technology writer and editor that has spent the last decade covering hacking, cyber threats, and information technology security, including senior positions as a writer, editor and industry analyst. His work has appeared on NPR’s Marketplace Tech Report, The Boston Globe,, Fortune Small Business, as well as ZDNet, Computerworld, InfoWorld, eWeek, CIO , CSO and He was, yes, a guest on The Oprah Show — but that’s a long story. You can follow Paul on Twitter here or visit his website The Security Ledger.

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.