05 May 2017

Managed Bug Bounties: Quality is in the Secret Sauce


By Jeff Cariker, Vulnerability Operations

Bug bounty programs have swept the security world by storm. As more companies are showing increasing willingness to let white-hat hackers probe their IT assets, we’ve gotten a lot more data around the efficacy and the results of these types of security practices. With public, private, 3rd party, and managed programs there’s a lot to consider in deciding which program fits the bill. Running Vulnerability Operations for Synack I work closely with a lot of customers. They tell me that they want to be thorough in finding vulnerabilities and need to make sure the solution fits into their daily operations.  It should help them communicate security issues to other groups within their company. So from the operations perspective, here’s my view on vulnerability reports, the significance of “Signal to Noise” ratio, and the importance of an efficiently managed bug bounty vulnerability program.

Something’s Off with Vuln Reports…

Many of the biggest tech companies are now running their own bug bounty programs, but I’ve noticed in public bug bounty statistics signal to noise ratio is extremely low. Some can be as low as 5%, but ranging up to mid-30%. Here a bug bounty company compared SNR among public companies with their own bug bounty programs. In terms of SNR, the results aren’t very impressive…

  • Facebook reported in 2015 that out of 13,233 submissions, they ended up with a total of only 526 valid reports- which is a mere 4% validity rate. And out of submissions from 5,543 researchers, only 210 researchers produced valid vuln reports.2
  • Github released a long-term overview of their bug bounty program to date. Over two years, 7,050 submissions came in, of which 1,772 warranted review, and only 102 total vulnerabilities (ranging from low to high risk) were paid out. The rate of total payouts to total submissions was only 1.4%.3

Signal to Noise Ratio (SNR)

The higher the “signal” is to “noise”, the better.

Generally, “signal to noise” refers to the ratio of useful information to false or irrelevant information. In the bug bounty world specifically, the “Signal to Noise Ratio” (SNR) is defined in terms of reported vulnerabilities: valid, original vulnerabilities vs invalid, duplicate, or false positive vulnerabilities. You want to find more valid reports in the bucket of overall submissions, because then it takes less work to comb through invalid reports that are practically useless to you. Let’s be honest: we’re all busy people and would much prefer less work as opposed to more. Where SNR is concerned, a high ratio means the bug bounty program is delivering better value to the company.


Uber wrote a report detailing the first 100 days of their bug bounty program. Out of 2,030 total reports that came in, the number of security vulns found and fixed totaled 161 and the Signal to Noise ratio was 1:6. In a later report, the company boasted an increased SNR of 1:5, which is still only a 20% efficiency rate when combing through thousands of vuln reports.1,4

How do you know what an invalid vulnerability submission looks like? The following are types of submissions my team filters out for our clients.

  • Low impact / Low-risk vulnerabilities
  • Vulnerabilities that are not reproducible
  • Vulnerabilities that are theoretical and provide no proof of exploitation
  • Duplicate vulnerabilities
  • Out of scope vulnerabilities (not the correct testing environment)

Any of these submission types can result in more noise than signal and contribute to a lower SNR. If you’re getting a low SNR with the vulnerability reports you receive, I’m afraid you are wasting your team’s precious time and resources.

High SNR

A high SNR means that your bug bounty program is efficient and effective

Almost every vulnerability received affects the business and will get fixed by your team. When you have high signal and low noise, it saves you the headcount needed to review the validity of vulnerability submissions which allows your team to focus more on critical matters.

How to Prevent Low SNR

  • Have a clearly defined scope that excludes every category that you consider to be low risk to your business and also all known vulnerabilities.
  • Uphold the low-risk and out-of-scope list: For the most part, no vulnerabilities submitted in the low and out-of-scope list should be accepted. To use all of your research talent efficiently, you should guide researchers to the highest priority targets and discourage them from wasting time on vulnerabilities that don’t really matter to you.
  • Make use of tools that easily detect duplicate vulnerabilities early – This could include a vulnerability tracking system where you can search by vulnerability and by URLs, or even a generic spreadsheet. Multiple researchers will often hone in on the same glaring vulnerabilities. To save time, identify duplicates quickly so that researchers can move on to new targets.
  • Fully reproduce every single vulnerability to ensure its quality and to uphold a high signal. Once the vulnerability is validated, pay researchers quickly to incentivize them to stay motivated and engaged.
  • Establish a direct line of communication with researchers to inform them quickly if a known vulnerability will not be fixed so they can refocus their efforts. Note: this cannot work in a public bounty.

The Synack SNR

Since Synack’s beginning, we’ve been dedicated to providing a high SNR to our customers, where our committed goal is >95% signal . We have always strived for this, and have made substantial improvements to our team and the way we process vulnerability reports. This included tweaking our low-risk and out-of-scope list, being more thorough in testing each and every vulnerability, and also sending vulnerability reports back to researchers for further clarification if needed.

Once the vulnerability operations team goes through the process above, the resulting SNR to the client is 98%, and in fact, we incorporate a lot of aspects into our process to ensure quality and consistency above all else.

The Recipe for Success

By Jenn Yonemitsu, Community Outreach

Application process:
For our standard* programs Synack has a stringent 5-stage vetting process and only considers researchers 18 years or older that have 3-5 years previous experience in penetration testing. This helps to ensure that vulnerability reports are from more experienced researchers and from a quality perspective helps provide you with an experienced viewpoint.

Scoping Process:
We’ve found it to be important to have a team to review the scoping of your system to determine the details of what to test, what is important, and anything that is considered too low risk. At Synack we highlight low risk vulnerability types in each target for both our researchers and you.

Scoping Service:
A scoping service can point the researchers to the in-scope targets and also highlight any areas that are out of scope to help researchers who want to import the scope of the penetration test into their tools. Our scoping service also provides details to our specialized vulnerability management system to flag any submissions that might be out of scope to our Vulnerability Operations team.

Duplication tools:
Dups impact everyone’s time, create extra work and frustrate researchers who put a lot of time into finding a vuln. We provide a feature we call Vuln Analytics which shows the researcher vulnerability categories and indications about where  vulnerabilities occurred. This helps them avoid submitting a report for a vulnerability that has already been accepted and provides insight into where a vulnerability may have been missed.

Vulnerability Reproduction:
Each vulnerability report is reproduced by our Vulnerability Operations team to ensure accuracy and the actual impact to you. Everything needed to reproduce the vulnerability is provided in each submission by our researchers.

Report Quality Control:

  • Each vulnerability report submitted by a researcher should be thoroughly checked for accuracy and reproduction steps.  Our Vulnerability Operations team provides feedback to researchers when our report standards are not met to ensure high quality in the reports we send to you.
  • Providing details for a vulnerability allows for it to be reproduced and helps you understand the vulnerability. We require details such as: description and impact, vulnerable location(s), CVSS score and recommended fix.
  • Holding researchers to reporting standards has shown to help researchers become better penetration testers over time.

Fast payment:
Once all of the above is complete, the vulnerability must be paid out to the researcher. By default, Synack manages all interactions with the researcher through our Vulnerability Operations team. This alleviates additional operational or fiscal burden on you, where our team collects your input then manages researcher interactions on your behalf before accepting a vulnerability. This efficient process enables us to average less than 21 hours from vuln submission to acceptance / payment for our researchers. It allows us to work efficiently with our researchers, prioritize vulns for you and reduce any risk of breach.

Consistent payments:
Payments for vulnerabilities are set based on its category for which we have a defined minimum and maximum award. Each category varies in the min/max value based on the impact to you. The payment amount will be between the min/max value and specifically calculated by a mixture of techniques, including the CVSS score and the subjective business impact. Both the CVSS score and business impact are verified and determined by our team for accuracy.

*In the spirit of helping to grow and develop new cybersecurity talent, we have a special program for students. The Synack platform is setup to provide different levels of management and structure to support this talent and foster safe hacking principals.




Customers tell us that every time they receive a vulnerability submission from us, they know it’s high-impact and that it’s critical to fix.

Eliminating the noise and providing a trusted, fully-managed security platform is something I’m very proud of at my company and the team we have built. Our customers appreciate that they don’t have to spend a lot of time trying to sift through vulnerability submissions to figure them out. Using a managed bug bounty program leads to a more efficient process for mitigating vulnerabilities inside a company which leads to more effective time management and the ability to have internal security teams focusing on more long-term growth projects.
There is a lot to be gained from utilizing a complete vulnerability management platform for your security processes. Not only do you save time and become more effective at patching vulnerabilities, you also get to go “under the hood” to see data and metrics that show how hardened your applications are against attack and how resilient they are across their attack surfaces. With all of this, you can build a more stringent defense against cyber attacks and effectively manage security risk for your business… the number one goal of the Synack “Secret Sauce”!


1. Bryant, Matt, Rob Fletcher & Collin Greene. (2016 August 11). 100 Days into Uber Engineering’s Public Bug Bounty Program [Blog Post] Retrieved from: https://eng.uber.com/bug-bounty-update/

2. Facebook Bug Bounty. (2016 February 9). Highlights: Less Low-Hanging Fruit [Blog Post] Retrieved from: https://www.facebook.com/notes/facebook-bug-bounty/2015-highlights-less-low-hanging-fruit/1225168744164016

3. Github. (2016 February 4). Two Years of Bounties [Blog Post] Retrieved from: https://github.com/blog/2099-two-years-of-bounties

4. Uber. (2017 March 22). Celebrating a Year of Smashing Bugs [Blog Post] Retrieved from: https://medium.com/uber-security-privacy/uber-bug-bounty-year-one-e0464bcfddd7