Securing Web Apps in a DevOps World (Notes From Black Hat 2017)

Zane Lackey of Signal Sciences spoke at Black Hat 2017 on a topic near and dear to my heart: Practical Tips for Defending Web Applications in the Age of DevOps.

DevOps — and really, any Agile or Agile-like rapid software development approach — is a huge enabler for business. Changes to software are envisioned, implemented, tested, and deployed incredibly fast. Deployments can happen multiple times per day. The agility this offers an organization is outstanding.

But DevOps is also a thorn in the side of traditional application security programs. The “gates” that most such programs rely on no longer exist. There are far fewer obvious places for a security team to hook into the development process. And that means there are more security bugs that escape to production.

It’s not all gloom, though: the rapid lifecycle also means that production security bugs, once discovered, can be fixed very quickly. As Lackey points out, DevOps models give us a chance to outpace attackers, potentially closing holes before they have a chance to be meaningfully exploited. In fact, he shared a story of a researcher who discovered a bug on Etsy’s platform on a Friday evening; before this researcher even had a chance to notify Etsy, they had noticed the attack, identified the problem, and deployed a patch. All by Saturday morning.

Lackey made a few specific recommendations for security programs in DevOps organizations:

Change how you use static analysis. A lot of organizations use static application security testing (SAST) as a way to identify and “ban” specific practices. This is good, as long as it can be automated on short timescales. But it’s also important to highlight where proper practice was followed, and flag inherently difficult-to-get-right operations (like crypto) for security engineering resources so that they can have discussions about whether this is the right solution, as opposed to only whether it’s been implemented correctly.

I’m not sure I agree about that last statement: I think having those validation (“did we make the right choices?”) discussions like that is better accomplished through building security champions within the dev teams. I’d much rather have a developer on the team with enough security knowledge to know when to engage the experts than rely on a tool to highlight it.

But as for the rest of it, I wholeheartedly agree. In fact, it made me smile to hear it in part because Veracode just released Veracode Static Analysis IDE Scan (an IDE-based, screaming fast SAST tool) exactly because we see how important it is to get rapid feedback to developers (within seconds), and tell them what they did right as well as what they might have got wrong.

Change how you use dynamic analysis. Lackey’s key complaint about dynamic application security testing (DAST) tools is that most struggle with modern-style web applications, since they involve a lot of client-side logic (e.g., single-page applications) and an API-centered design approach that many DAST tools can’t understand. (I confess I had a moment of pride, because I think Veracode does pretty well here — and gets better all the time!).

But also, and I think more importantly, too many security organizations use DAST as a penetration test replacement. That is, it’s done occasionally, on a schedule, and with the goal of finding as many potential problems as possible. Instead, Lackey recommends using DAST continuously, and primarily to enforce configuration policy (e.g., do you have a proper TLS configuration?), and guard against old flaws being reintroduced.

Increase security visibility. One of the key lessons of DevOps has been the importance of breaking down operational data silos. DevOps is doomed in organizations that don’t understand the importance of getting all relevant operational data (logs, alerts, performance metrics, etc.) into one place for everyone to understand and respond to. If security teams are to participate with DevOps teams in making applications safer, they need to participate in breaking down the data silos.

The DevOps teams need to be able to have security data visible, so that they have proper context for their decision-making. I suspect this will be one of the harder things for security teams to do: we tend not to want to give up control.

Improve security feedback. The main sources of security feedback today are malicious attacks and the annual penetration test results. This simply isn’t fast enough to be useful on its own in a DevOps organization. Adding in a well-run bug bounty program allows for a better path to continuous testing and feedback.

I would add to this advice that continuous use of SAST and DAST tools, as well as internal red teaming, can provide additional sources of security feedback that are invaluable to DevOps teams.

Continuous improvement. Exercising these more-agile security processes should lead to constant improvements. Building an ability to detect and respond to attackers ever earlier in the attack path, and refining incident and flaw response processes, during normal operation creates a capability to not only prevent more attacks, but to respond quickly and effectively when attacks occur.

In all, DevOps is shaking up the traditional security model, and security teams need to evolve and participate. And as always, build efficient and effective processes around carefully-chosen tools.

Darren is a Security Researcher at Veracode, passionate about AppSec advocacy and education. He also enjoys making stuff, coffee, and weirdness.