Live From Black Hat USA: The Inevitable Marriage of DevOps & Security

During her briefing with Kelly Shortridge, vice president of product strategy at Capsule8, Dr. Nicole Forsgren, research and strategy at Google, did a beautiful job of adding imagery to the story she told of the attendee reactions during the now-famous talk Paul Hammond and John Allspaw gave at Velocity in 2009. If you're not familiar, the title of said talk was, "10 Deploys Per Day: Dev & Ops Cooperation at Flickr."

Forsgren recalled that, "The room was split. At the end of this process, large pieces of code would be deployed and, basically, lit everyone on fire. Half the room was amazed and it was changing the world. Half of the room said they were monsters and how dare they light people on fire 10 times per day." Forsgren concluded that "DevOps has crossed the chasm - the business benefits are too striking. We see most of the industry doing this. There is no turning the ship around."

Indeed, DevOps has long moved beyond the conceptual and has become a widely adopted practice in software development and delivery. It gave birth to the InfoSec equivalent of DevSecOps and the concept of "shifting security left." From where I sit within Veracode, I see the ways that many security solutions providers are doing their best to provide developers with the tools they need to embed security into their workflow, yet it’s clear that there is still more to be done to get InfoSec professionals on board.

"James Wickett has said the ratio of engineers in development, operations, and InfoSec in a typical technology organization is 100:10:1. If we integrate [InfoSec professionals] earlier to have input, the shift left can build a more collaborative culture, contribute to amazing outcomes - like stability, reliability, and resiliency," Forsgren said. "We need to build secure systems, and we will find ways to do this. We know this is super important, and security is the next frontier. Security can contribute to this and join DevOps. Or you can stand aside as DevOps figures this out and carves its own path. I would love to see InfoSec contributing the expertise we just don't have."

Forsgren was clearly echoing the sentiment Dino Dai Zovi expressed in his conference keynote. Certainly, the concept of being lit on fire 10 times per day would create a fight-or-flight response, and it is much easier to go to no than to go to yes. Yet, when Forsgren spoke about the benefits of this type of work, she explained that what InfoSec pros would face would be mini-fires with a smaller blast radius. She argues that it is time for InfoSec to say, "no, and…"

The Security of Chaos

It appeared that Shortridge couldn't have agreed more.

"The real DevOps will be held accountable for security fixes," said Shortridge. "So what should goals and outcomes become? Why should InfoSec and DevOps goals diverge? InfoSec should support innovation in the face of change - not add friction. InfoSec has arguably failed, so 'this is how we've always done it' is invalid. The greatest advances in security are rarely spawned by the security industry."

In other words, it's time to start jumping out of the proverbial planes in order to face our fears and start doing things differently in security. Shortridge reminded us that it is inevitable that things will fail and things will be pwned, which is why she is a proponent of adopting chaos engineering. Chaos engineering is the discipline of experimenting on a software system in production to provide your organization with a level of confidence in the system's capability to withstand turbulent and unexpected conditions, while still creating adequate quality of service (resiliency) during difficult times.

The concept of chaos engineering was created while Greg Orzell was overseeing Netflix's migration to the cloud in 2011. He wanted to address the lack of adequate resilience by creating a tool that would cause breakdowns in their production environment - the one used by Netflix customers. In doing this, the team could move from a development model that assumed no breakdowns to one where they were considered inevitable. This encouraged developers to build resilience into their software from the start. By regularly "killing" random instances of software service, they could test redundant architecture to make sure that a server failure wouldn't noticeably impact the customer experience.

"Expect your security controls will fail and prepare accordingly. System architectures must be designed assuming the controls and users will fail," she said. "Users very rarely follow the ideal behaviors. Don’t try to avoid incidents. Embrace your ability to respond to them. Ensure that your systems are resilient enough to handle incidents gracefully. Pivot toward realistic resilience."

If your team can plan for nothing but the chaos factor, then you should understand that there are true benefits to applying chaos resilience, including lower remediation costs, decreased stress levels during real incidents, and less burnout.

"Incidents are a problem with known processes, rather than fear and uncertainty. It creates feedback loops to foster understanding of systemic risk. Chaos engineering does this to help us continuously refine security strategy - essentially all the time red teaming. You have the ability to automate the toil, or the manual, repetitive, tactical work that doesn't provide enduring value," she said.

How to Marry DevOps and Security

At the end of the talk, Forsgren offered these tenants for a scalable love between DevOps and Security:

1. Sit in on early design decisions and demos – but say “No, and…” vs. “No.”
2. Provide input on tests so every testing suite has InfoSec’s stamp on it.
3. By the last “no” gate in the delivery process, nearly all issues will be fixed.
4. InfoSec should focus on outcomes that are aligned with business goals.
5. Time To Remediate (TTR) should become the preliminary anchor of your security metrics.
6. Security- and performance-related gamedays can’t be separate species.
7. Cultivate buy-in together for resilience and chaos engineering.
8. Visibility/observability: collecting system information is essential.
9. Your DevOps colleagues are likely already collecting the data you need - work with them to collect it.
10. Changing culture: change what people do, not what they think.

Forsgren and Shortridge made the case that security cannot force itself into DevOps, it must marry it - and have an equal partnership. Chaos/resilience are natural homes for InfoSec and represent its future, and InfoSec will need to evolve to unify responsibility and accountability.

"If not, InfoSec will sit at the kids’ table until it is uninvited from the business," Shortridge said. "Giving up control isn’t a harbinger of doom. Resilience is a beacon of hope."

Stay tuned for more from Black Hat …