(Editor’s Note: This blog post is based on a study conducted in 2017. A follow-up Pentest as a Service Impact Report, published in 2020 and also written by Dr. Wang.
Chenxi Wang, Ph.D, is a tech entrepreneur and investor who is passionate about privacy and security. She was named by SC Magazine as a 2016 Women of Influence. She founded Rain Capital in 2018 where she is managing partner and she sits on many boards at startups and VC funds. She is the former Chief Strategy Officer of Twistlock. She also is an alumni of Forrester Research, Intel Security and CipherCloud. She holds a Doctor of Philosophy in Computer Science from the University of Virginia.
Why do security metrics matter?
Here’s a metric-related question that every application security leader should be able to answer:
What value are we getting from our investments in our application security program?
One reason this is a challenge is that there are many options for a given application security control that a security or engineering leader may want to implement. For example, when it comes to application security testing, there are automated vulnerability scanners, security consultants, bug bounties, and crowdsourced penetration testing (pentesting).
Although security consultants are involved for both Pentest as a Service (PtaaS) and traditional pentesting, each option has unique associated penetration testing costs and benefits. Return on Investment (ROI) factors to consider include:
-
Pentesting cost
-
Staff time spent on vulnerability triage
-
Staff time spent to recreate and validate vulnerabilities
-
Staff time spent managing pentesters
-
Duration of testing
Overview of My Research into Pentesting ROI
I have always been interested in metrics, and in particular how accurate metrics can help security and engineering leaders make better decisions that have to do with investment of money, time, and resources.
I recently became interested in measuring the efficacy of one’s application security program. I partnered with Cobalt on a study to specifically investigate the Return on Investment for pentesting. More specifically, I wanted to understand the associated costs and benefits of Pentesting as a Service (PtaaS), as compared to the traditional method of pentesting — hiring a team of professional pentest consultants (also called white hat hackers or security researchers).
As part of this study, I interviewed many organizations that are using PtaaS, and compared their experience with traditional pentesting services and penetration test pricing. I found that the ROI of a PtaaS engagement is 96% higher than traditional pentesting, primarily due to increased accuracy, lower penetration testing services prices, and improved efficiency.
1. Reduced Pen Testing Cost
Let’s start with an obvious question: How much does penetration testing cost, comparing the two approaches? Based on a series of deep-dive interviews with customers who buy pentest services , I found a significant impact on penetration test pricing. Pentesting as a Service reduced the cost of penetration testing by an average of 31% compared to similar testing conducted by traditional pentesting consultancies.
In other words, the approach matters significantly in penetration testing pricing. A traditional pentest usually ranges from $20,000-$50,000. Of course, many factors, including scope, can impact price. Research has shown that pentesting as a service is approximately 31% less expensive, so a $20,000 engagement, for example, might cost $13,800 with a pentest as a service approach. Another way to think of it, according to Osterman research, “the number of person-hours spent managing tests drops from 7.5 person-hours with traditional consultancies to 2.8 with Cobalt, saving clients around $415.”
2. Improved Triage Efficiency and its Effect on Cost of Penetration Testing
In pentesting, triage is the in-house effort of understanding the criticality, context, and nature of each vulnerability and then prioritizing its remediation compared to other vulnerabilities and organizational initiatives.
In a PtaaS model, the client’s team can streamline time-consuming triage processes that used to require many back and forth meetings and calls between security engineers and engineering (or between engineering and testers). On average, PtaaS helped organizations reduce triage time to approximately 20 minutes per vulnerability (compared to 89 minutes per vulnerability with traditional pentesting services).
This translates to an average saving of approximately 29 hours per test, assuming that an average test uncovers 20 to 30 vulnerabilities or issues in the high or medium category.
For a company that conducts semi-annual assessments on a large application portfolio, the saving can be significant. For example, for an organization with 100 applications, this could provide a savings of 5,800 hours of staff time or three full-time positions. Assuming 235 work days per year, the average cost of a fully burdened security engineer is $167,000, funds that can be well spent elsewhere. Staff support costs should be considered when an organization considers the question: “How much does a penetration test cost?”
3. Increased Test Depth and Coverage
80% of the companies that deployed traditional pentesting services reported shallow findings and limited test coverage by those testers. This is a source of great concern. Many of these companies told me that in order to augment the shallow tests and obtain the insight required to correct a flaw, they ended up spending a good 20–30 hours of in-house personnel time for further testing and validation. With PtaaS, customers reported an increased test depth, a broader range of attack techniques, more extensive test coverage, and less non-trivial time spent by in-house staff on supplemental testing saving augmented testing by in-house staff. The Director of Engineering of an enterprise software company said, “The Pentest as a Service team crafted some inventive attacks. One test involved abusing of the business logic in a way that none of us had seen before. It was educational even for my developers.”
4. Decreased Management Burden
With a pentest consultancy, sometimes the customer has daily calls with the security researchers in order to keep on top of the progress. One interviewee , the InfoSec Officer for a financial technology company, said that “waiting for a daily call to occur and dealing with the added overhead of a call every day is challenging.” With PtaaS, the officer manages the project and keeps up with the team by simply responding to notifications from the platform. Dashboard views give a visual pulse check on the progress of testing.
On average, the interviewees said they were spending 7.5 hours managing projects with traditional pentesters. With PtaaS, that number drops to 2.8 hours.
5. Shortened Time to Results
With a traditional pentest consultancy, the testers conduct their tasks in a “black box” — the customer has little visibility into what is going on until he or she receives a pentest report that describes the findings. With PtaaS, the tests performed and vulnerabilities discovered are documented in the platform as testing is performed and while security researchers are readily available to support triage and remediation. Security or development teams can log onto the PtaaS platform and see real-time information on the ongoing tests. Assessment results are visible in the platform as soon as they become available — no need to wait for a final report. While PtaaS requires a bit more time in the beginning to set up and gather information, the time to first result is shortened from a minimum of 2 weeks with a pentest consultancy to a day or sometimes hours with PtaaS. On average, time to final results is shortened from 3.1 weeks with the consultancies to 2.25 weeks with PtaaS, an improvement of .85 weeks.
The biggest impact of reducing the time to results is the reduced window of exposure for vulnerabilities. With triage and remediation potentially starting .85 to 3.1 weeks sooner, attackers have less time to exploit discovered weaknesses. By releasing results in real-time, organizations can in many cases fix vulnerabilities before the next build. In many cases, the client flags vulnerabilities as ready for retest in the platform before the initial pentest is completed. In other words, PtaaS better keeps pace with DevOps and SecOps than traditional pentesting.
In Conclusion
Clearly, ROI goes beyond answering the simple question: How much does a pentest cost?
To summarize, I found that the ROI of Pentesting as a Service is 96% higher than traditional pentesting. When compared to traditional pentests, PtaaS:
-
Costs 31% less
-
Reduces staff time spent by 69 minutes of triage per vulnerability, which is an average of 29 hours per pentest
-
Requires up to 30 hours less in-house effort to recreate and validate vulnerabilities
-
Reduces the staff time spent managing pentesters by 4.7 hours per test
-
Delivers findings .85 to 3.1 weeks sooner, which reduces the time period attackers could exploit weaknesses through hacking
These conclusions are based on in-depth interviews with organizations that are using PtaaS. The companies I interviewed represent a wide swath of industry segments, including SaaS, enterprise software, healthcare, and FinTech.
What does this mean for your organization? Consider conducting your own analysis to compare different application security options, determine the penetration testing costs and benefits for your organization, and find the right model to deliver the best Return on Investment.
With PtaaS increasing in popularity, pentesting can better align with the pace of your organization while increasing efficiency, decreasing costs, and making your organization more secure. While this paper focused on general application security, these new solutions are available for web applications, mobile applications, APIs, networks, AWS environments, systems, or other assets and resources.
For more information, read the free Economic Report on penetration testing with Cobalt's Pentest as a Service (PtaaS) platform.
