One of the counterintuitive features of DevOps culture is a willingness to fail. In our success-oriented culture, this might sound like exactly the wrong direction in which to take your development teams. But a willingness to fail quickly, and often, can paradoxically lead your teams to greater success — provided you do it in a structured way and you learn from your failures.
There’s a lot of good articles out there about why “fail fast, fail often” is key for DevOps — and why it’s a counterintuitive concept, particularly in some international cultures. It’s not particularly well understood with a lot of commentators in the business press saying “Don’t fail fast! Iterate instead!” — completely missing the point. When we talk about “fail fast” in DevOps, iterating is exactly what we’re talking about.
You can realize the most value from “fail fast” by failing planfully and institutionalizing learning from failure. In product development, this might be accomplished by doing hypothesis-driven development — create a hypothesis about user behavior, test in a controlled, low-cost experiment, and go forward with the learnings. In product deployment, this is what canary deployment is all about. If your new architecture isn’t going to perform well, better to figure that out before you deploy it to your whole user base, and roll it back.
“Fail fast” meshes well with the “three ways of DevOps”: systems thinking, amplifying feedback loops, and creating a culture of continual exploration and learning. In addition to using “fail fast” to catch individual security defects, applying other best practices like blameless post mortems can help reinforce the learning culture.
There are two key areas in the software development lifecycle where “fail fast” is most beneficial:
1. Developers can run static analysis on changed components in a private developer sandbox before checking into the pipeline, which is proven to help developers fix more vulnerabilities.
2. Being able to get application security testing results from instrumenting a live application during integration testing is better than waiting until just before — or just after — release. Doing static analysis on the whole application via an automated scan from Jenkins, Visual Studio Team Services, or Ansible before deploying to the integration testing environment is even better.
All this testing is good, but to drive improvement, developers need tools to learn the lessons from the findings, so they can fix the vulnerabilities quickly. Providing actionable results is step one. Giving developers the know-how to address vulnerabilities via just-in-time secure development training and remediation coaching is even more important. (Don’t take my word for it; look at the numbers in the latest State of Software Security report, which shows substantial benefits in fix rate when developers use sandboxes, eLearning, and remediation coaching.)
At Veracode, “fail fast” is an essential part of our philosophy of securing DevOps. Check out our free guide to learn about how “fail fast” fits into our Five Principles for Securing DevOps.