Before a company begins a new project or program, it’s important to determine what success for that project means. Setting metrics and goals based on the program’s objective can help companies and departments evaluate the success of a program or provide insight into where adjustments should be made. While metrics and key performance indicators are at the center of a performance-based culture, many organizations continue to miss the mark by measuring the wrong things or focusing on aspects of a program that don’t truly measure its health.
The stakes of measuring the wrong aspects of your program are even greater when it comes to application security. Amid a fast-changing and increasingly perilous landscape, a lack of visibility and understanding can undermine or even cripple your enterprise, and open the door to breaches, breakdowns, and stolen data — all while you believe your program is working because your KPIs indicate success.
Metrics — or perhaps more accurately, the right metrics — are crucial for understanding what’s really happening in your AppSec program. They serve a dual purpose: They demonstrate where your organization is at but also show what progress it’s making in achieving its objectives.
The key to a truly effective AppSec program is to identify the right metrics and present them in a way that your executives and key business decision-makers can understand and use. An overly cumbersome approach, or one that presents information in an obtuse and burdensome way, isn’t likely to succeed.
This means devising a program that supplies the right metrics — and in some cases the right single metric — to a decision-maker at the specifc time and place it’s needed. For most companies, this task revolves around two overarching factors: the total number of apps in your program and the percentage of apps that are in compliance with your AppSec policy. Together, these criteria allow your organization to drill down and understand an array of factors and conditions that can make or break your cybersecurity initiative.
Measuring policy compliance is a critical aspect of an effective application security program.
It’s key to focusing your investments and resources, quantifying your progress, and understanding your risk profile.”
Principal Product Manager, Reporting and Analytics, Veracode
Application security presents a few unique challenges. First, today’s software landscape — from the products themselves to development methodologies to the types of threats — is constantly changing and evolving. Add in DevSecOps and other Agile initiatives, and the challenges grow, sometimes exponentially. Second, the number of applications and the volume of code that organizations must manage are continually growing. All of this introduces new and different threats, particularly as organizations increasingly take advantage of open source libraries to speed their software development. Third, amid an overwhelming barrage of cybersecurity threats, AppSec is only part of the picture — it’s just one piece of your overall cybersecurity program.
Yet, application security doesn’t have to be an overly complicated or burdensome task. The right set of metrics and key performance indicators (KPIs) can greatly simplify and streamline both your software development and your application security. SANS Institute points out that AppSec metrics fall into three major categories: technical, operational, and executive.1 Organizations that use the right AppSec metrics and make them available to the right people channel resources more effectively and improve code quality. This has positive repercussions beyond the four walls of your organization: It reflects outward to your business partners and customers, who make decisions based on the quality and security of your code.
1 Benchmarking AppSec: A Metrics Pyramid. SANS Institute
By using metrics and tangible data to support an application security initiative, you can take performance to a higher level. Find out how by reading Building a Business Case for Expanding Your AppSec Program.
Your organization likely conducts threat modeling and risk ranking to understand where to focus scarce security resources. Your application security policy should stem from a similar exercise, starting with looking at your entire application inventory and assigning groups of applications different risk categories. Develop these categories by asking simple questions such as:
Based on those answers, you can determine which scan frequency and testing types are required, as well as which types or severities of flaws may be acceptable. Perhaps disallowing the OWASP Top 10 is sufficient to meet your security needs, or maybe you want to disallow the SANS 25 or all very high severity findings. You should also be stipulating the appropriate testing techniques for each grouping of applications. Internet-facing applications likely require dynamic analysis in addition to static analysis. Also consider the frequency of testing for each group of applications. Perhaps an annual manual penetration test is required, both to meet your PCI requirements as well as your risk ranking.
It's important to note that you shouldn’t develop the security policy for your organization in a vacuum, but rather in conjunction with software development. Working with development teams to ensure that the security policy is both addressing the risk for the organization and achievable will help ensure adoption overall.
Organizations, particularly as they move to Agile and DevSecOps, should find themselves scanning applications and code more frequently. And it’s critical for these scans to occur at the right time and within the right workflow that aligns best with the development team. When scans are out of alignment with processes, the risk of introducing vulnerabilities increases. Critical metrics revolve around tracking when scanning is occurring: Is it with each release or on a schedule (daily, monthly, yearly)? In addition, is scanning taking place in an ad hoc fashion or through automated integration with development systems?
Often, the applications for different development teams, business units, or groups need to be compared in order to understand overall risk and focus application security resources in the most impactful location. However, comparing applications can be challenging. Policy compliance as a metric may help here, but if both groups of applications are failing policy, where is the greater risk?
Further, the prevalent categories or types of flaws differ from language to language; a MicroService written last week that’s 1MB differs greatly from a legacy stem written fve years ago. So how do you compare apples and elephants? Flaw density provides a way of looking at the number of flaws produced from a static analysis over the size of the application and can provide directional guidance when comparing groups of applications. A high flaw density simply means more flaws to address. You could choose to focus only on very high and high severity flaws for your flaw density metrics and better compare sets of applications. This comparison would give managers a sense of where to use application security resources — such as training or working with remediation consultants — to have the biggest impact for your business.
An AppSec program is only as good as the flaws it actually fixes. If your development teams are addressing fixes quickly and effectively, your organization’s risk exposure is likely to shrink. If the fix rate falls short, however, your development teams may need to adjust. For instance, your organization may require new or better training methods, or it may need to adapt workflows and processes to better address deficiencies. For open flaws, it’s important to understand your “high” and “very high” severity findings. This will allow your organization to direct resources to where they’re most urgently needed.
Learn more about the metrics that can help you prove your applications are secure and that your AppSec program is making a difference in our webinar, “Application Security Metrics & How to Track Success.”
Integrating metrics into workflows and processes is critical. According to OWASP, organizations must address several important questions in order to build a strong metrics framework.2 These questions revolve around three areas: application security process metrics, application security risk metrics, and software development lifecycle (SDLC) security metrics.
Get details on integrating security into the software development lifecycle in this Securosis report, Building an Enterprise DevSecOps Program.
A holistic and comprehensive application security program revolves around both high-level metrics and supporting metrics. It involves also people, processes, and technology. When all these factors are aligned — and when executives, line of business managers, and software developers are connected and tuned into the specific requirements of the organization — AppSec evolves from a general concept to a powerful tool. A metrics-centric AppSec program is particularly valuable in today’s business environment, where Agile, lean, and DevSecOps rule. A core tenant of each of these is to measure/make work visible. In order for application security to truly be embraced by software development, follow this philosophy and provide clear visibility into how the program is running. Ultimately, the right metrics can help your organization reduce risk but also achieve a competitive advantage through higher-quality software code. The right metrics also reduce costs associated with security and aid in an organization becoming more proactive. This, in the end, generates both top-line and bottom-line gains.
We’ve helped thousands of companies, large and small, develop AppSec programs that work for them, and we can help you.
Contact Us for help getting started or for moving to the next step.
Veracode is the leading AppSec partner for creating secure software, reducing the risk of security breach and increasing security and development teams’ productivity. As a result, companies using Veracode can move their business, and the world, forward. With its combination of automation, integrations, process, and speed, Veracode helps companies get accurate and reliable results to focus their efforts on fixing, not just finding, potential vulnerabilities. Veracode serves more than 2,500 customers worldwide across a wide range of industries. The Veracode cloud platform has assessed more than 14 trillion lines of code and helped companies fix more than 46 million security flaws. Learn more at veracode.com, on the Veracode Blog, and on Twitter.
Copyright © 2021 Veracode. All rights reserved. All other brand names, product names, or trademarks belong to their respective holders.