Google went to great lengths to educate their developers about the benefits of security testing - even developing educational materials specifically to be read on the toilet.
Back in June, Mike Bland, a former Google Engineer, penned a great blog post that provides something of a roadmap to implementing a culture of secure development and testing. Bland worked at Google from 2005 to 2011 and, for much of time, was Google's "Testing Evangelist," helping to implement a system for thorough application testing and, more importantly, making practices like unit testing for developed code a cultural norm at Google. Bland's post is worth printing out and reading. He starts off with an analysis of the Apple GoTo Fail and Heartbleed vulnerabilities, and how they're object lessons for the importance of unit testing. (Veracode tackled some of the same issues regarding Goto Fail in this blog post.) But Bland also wraps in a (substantial) guide to developing a culture of secure development and testing. Bland weighs in on the relative importance and merits of various types of tests - integration testing vs. unit testing vs. fuzzing. And he talks about the organizational challenges of growing a culture of testing at Google. Bland says that, contrary to what you may believe, Google's engineering-heavy culture was not hospitable soil for the adoption of a culture of unit testing. Despite the company's considerable resources, Bland notes that "vast pools of resources and talent" within the company often got in the way by "reinforc(ing) the notion that everything is going as well as possible, allowing problems to fester in the long shadows cast by towering success." "Google was not able to change its development culture by virtue of being Google," Bland writes. "Rather, changing Google's development culture is what helped its development environment and software products continue to scale and to live up to expectations despite the ever-growing ranks of developers and users." Bland describes an all-hands approach to bending Google's engineering practice in the direction of more testing. Resistance, he said, was the byproduct of complex forces: a lack of proper developer education in unit testing practices and "old tools" that strained to scale with the pace of Google's development. Bland's response was to form what he calls a "Testing Grouplet" within Google that served as a support group and community for like-minded folks who wanted to implement unit testing procedures. That group operated like testing guerillas, developing and driving the adoption of new tools that made unit testing less painful, sharing best practices and engaging in testing propaganda within the development organization. One of the most successful initiatives was "Testing on the Toilet," a circa 2006 program in which Grouplet members designed short (one page) lessons on a variety of topics ("Don't Put Logic in Tests," "Debug IDs") then plastered hundreds of Google bathroom stalls worldwide with the *ahem* tactical reading material. The idea here isn't to shock folks. Rather: its to take the role of education in your secure development program seriously. Bland said the bathroom literature was part of a multi-pronged education effort that also included more common elements like brown bag lunches and a guest speaker series. Check out Mike's full post on building a unit testing culture at Google here.