By Jeff Cariker, Vulnerability Operations
Bug bounty programs have swept the security world by storm. As more companies are showing increasing willingness to let white-hat hackers probe their IT assets, we’ve gotten a lot more data around the efficacy and the results of these types of security practices. With public, private, 3rd party, and managed programs there’s a lot to consider in deciding which program fits the bill. Running Vulnerability Operations for Synack I work closely with a lot of customers. They tell me that they want to be thorough in finding vulnerabilities and need to make sure the solution fits into their daily operations. It should help them communicate security issues to other groups within their company. So from the operations perspective, here’s my view on vulnerability reports, the significance of “Signal to Noise” ratio, and the importance of an efficiently managed bug bounty vulnerability program.
Something’s Off with Vuln Reports…
Many of the biggest tech companies are now running their own bug bounty programs, but I’ve noticed in public bug bounty statistics signal to noise ratio is extremely low. Some can be as low as 5%, but ranging up to mid-30%. Here a bug bounty company compared SNR among public companies with their own bug bounty programs. In terms of SNR, the results aren’t very impressive…
- Facebook reported in 2015 that out of 13,233 submissions, they ended up with a total of only 526 valid reports- which is a mere 4% validity rate. And out of submissions from 5,543 researchers, only 210 researchers produced valid vuln reports.2
- Github released a long-term overview of their bug bounty program to date. Over two years, 7,050 submissions came in, of which 1,772 warranted review, and only 102 total vulnerabilities (ranging from low to high risk) were paid out. The rate of total payouts to total submissions was only 1.4%.3
Signal to Noise Ratio (SNR)
The higher the “signal” is to “noise”, the better.
Generally, “signal to noise” refers to the ratio of useful information to false or irrelevant information. In the bug bounty world specifically, the “Signal to Noise Ratio” (SNR) is defined in terms of reported vulnerabilities: valid, original vulnerabilities vs invalid, duplicate, or false positive vulnerabilities. You want to find more valid reports in the bucket of overall submissions, because then it takes less work to comb through invalid reports that are practically useless to you. Let’s be honest: we’re all busy people and would much prefer less work as opposed to more. Where SNR is concerned, a high ratio means the bug bounty program is delivering better value to the company.
Uber wrote a report detailing the first 100 days of their bug bounty program. Out of 2,030 total reports that came in, the number of security vulns found and fixed totaled 161 and the Signal to Noise ratio was 1:6. In a later report, the company boasted an increased SNR of 1:5, which is still only a 20% efficiency rate when combing through thousands of vuln reports.1,4
How do you know what an invalid vulnerability submission looks like? The following are types of submissions my team filters out for our clients.
- Low impact / Low-risk vulnerabilities
- Vulnerabilities that are not reproducible
- Vulnerabilities that are theoretical and provide no proof of exploitation
- Duplicate vulnerabilities
- Out of scope vulnerabilities (not the correct testing environment)
Any of these submission types can result in more noise than signal and contribute to a lower SNR. If you’re getting a low SNR with the vulnerability reports you receive, I’m afraid you are wasting your team’s precious time and resources.
A high SNR means that your bug bounty program is efficient and effective
Almost every vulnerability received affects the business and will get fixed by your team. When you have high signal and low noise, it saves you the headcount needed to review the validity of vulnerability submissions which allows your team to focus more on critical matters.
How to Prevent Low SNR
- Have a clearly defined scope that excludes every category that you consider to be low risk to your business and also all known vulnerabilities.
- Uphold the low-risk and out-of-scope list: For the most part, no vulnerabilities submitted in the low and out-of-scope list should be accepted. To use all of your research talent efficiently, you should guide researchers to the highest priority targets and discourage them from wasting time on vulnerabilities that don’t really matter to you.
- Make use of tools that easily detect duplicate vulnerabilities early – This could include a vulnerability tracking system where you can search by vulnerability and by URLs, or even a generic spreadsheet. Multiple researchers will often hone in on the same glaring vulnerabilities. To save time, identify duplicates quickly so that researchers can move on to new targets.
- Fully reproduce every single vulnerability to ensure its quality and to uphold a high signal. Once the vulnerability is validated, pay researchers quickly to incentivize them to stay motivated and engaged.
- Establish a direct line of communication with researchers to inform them quickly if a known vulnerability will not be fixed so they can refocus their efforts. Note: this cannot work in a public bounty.
The Synack SNR
Since Synack’s beginning, we’ve been dedicated to providing a high SNR to our customers, where our committed goal is >95% signal . We have always strived for this, and have made substantial improvements to our team and the way we process vulnerability reports. This included tweaking our low-risk and out-of-scope list, being more thorough in testing each and every vulnerability, and also sending vulnerability reports back to researchers for further clarification if needed.
Once the vulnerability operations team goes through the process above, the resulting SNR to the client is 98%, and in fact, we incorporate a lot of aspects into our process to ensure quality and consistency above all else.
Customers tell us that every time they receive a vulnerability submission from us, they know it’s high-impact and that it’s critical to fix.
Eliminating the noise and providing a trusted, fully-managed security platform is something I’m very proud of at my company and the team we have built. Our customers appreciate that they don’t have to spend a lot of time trying to sift through vulnerability submissions to figure them out. Using a managed bug bounty program leads to a more efficient process for mitigating vulnerabilities inside a company which leads to more effective time management and the ability to have internal security teams focusing on more long-term growth projects.
There is a lot to be gained from utilizing a complete vulnerability management platform for your security processes. Not only do you save time and become more effective at patching vulnerabilities, you also get to go “under the hood” to see data and metrics that show how hardened your applications are against attack and how resilient they are across their attack surfaces. With all of this, you can build a more stringent defense against cyber attacks and effectively manage security risk for your business… the number one goal of the Synack “Secret Sauce”!
1. Bryant, Matt, Rob Fletcher & Collin Greene. (2016 August 11). 100 Days into Uber Engineering’s Public Bug Bounty Program [Blog Post] Retrieved from: https://eng.uber.com/bug-bounty-update/
2. Facebook Bug Bounty. (2016 February 9). Highlights: Less Low-Hanging Fruit [Blog Post] Retrieved from: https://www.facebook.com/notes/facebook-bug-bounty/2015-highlights-less-low-hanging-fruit/1225168744164016
3. Github. (2016 February 4). Two Years of Bounties [Blog Post] Retrieved from: https://github.com/blog/2099-two-years-of-bounties
4. Uber. (2017 March 22). Celebrating a Year of Smashing Bugs [Blog Post] Retrieved from: https://medium.com/uber-security-privacy/uber-bug-bounty-year-one-e0464bcfddd7