CISO discussion at the Synack Suite - Black Hat 2019
05 September 2019

3 Real Questions About Crowdsourced Security Vendors You Should Be Asking Yourself as a CISO


A Discussion with the CISOs of Domino’s and Creative Artists Agency

Security has been straddling two worlds, one of tradition and compliance and the other of innovation and pragmatism. But innovation is quickly taking over as security practices are becoming smarter and more evolved in order to keep up with the changing digital and threat landscape. What’s important to consider today is different than what was important even just three years ago. Security is changing so quickly that it can be difficult to even know the right questions to ask when it comes to evaluating a solution.

We know that facilitating unfiltered, “real talk” among CISOs is more important than ever in this industry. With increasing pressure from boards and the public to see proof that security programs are actually working, CISOs are on the lookout for innovative solutions that can actually make an impact. Recently, Synack brought together CISOs in our network and stepped out of the way so they could have an honest conversation about what’s working today in security. For the third year in a row during Black Hat, Synack hosted an intimate lunch at our Delano Penthouse Suite to do just that. Attendees got real perspectives from fellow CISOs on their priorities in leading their companies to protect against adversaries.

CISO discussion at the Synack Suite - Black Hat 2019

Among others, Security Manager Ron Ulko from Domino’s and CISOs Jeff Blair from Creative Artists Agency answered questions posed by attendees in an open forum. One thing all CISOs agreed on – crowdsourced security testing should be a priority. Done right, this innovative approach offers much more ROI than traditional penetration testing or bug bounty. But in terms of how to find the right vendor, they had some advice to give in response to 3 major questions that stood out in the discussion.

  1. How should I evaluate a crowdsourced vendor?
    With many crowdsourced security vendors now claiming that they do pen testing, it’s hard to differentiate and understand what to look for. One CISO talked about vetting through a platform approach being a key differentiator – “Between the diligence they go through and the vetting, Synack went into so much detail from my perspective. Bug bounty vendors are still a bunch of guys scattered around the place. What Synack gave me was so much more controlled, with the flexibility of picking hackers based off skill and location…that’s attractive.” He talked about how “the beauty of the platform is you can go back and see what a researcher has done line by line – you get the intelligence.” Jeff Blair from CAA talked about the platform and the research funnel being the main differentiator when he chose Synack. “The others still had researchers willing to test apps, but coming from their own systems. The others were really just more managed bug bounty versus a true enterprise caliber pen test.” It’s important for a CISO to maintain awareness on the vendor’s vetting of the crowd, the vendor’s roadmap, and the amount of control a vendor offers. With high quality vetting comes high quality results. Synack’s signal-to-noise is 99.98%, providing security teams with efficiency in being able to prioritize remediation. This is done through an optimal combination of our smart technology platform and high quality Synack Red Team. Crowdsourced vendors are now building out technology to be able to augment the human testers.
  2. What aspects of control should I be looking for and focusing on in a crowdsourced security test?
    Many crowdsourced security tests are not transparent – security teams are unable to track what is being tested, by whom, and at what time. They’re unable to see how their assets are affected in real-time by security testing. Jeff Blair from CAA talked about the level of visibility with Synack has allowed his team to flip a switch and stop a pen test on-demand. Transparency through a platform is key to ensuring customers still maintain control over a crowdsourced pen test, especially as ethical hackers start to test IPs. Customization and flexibility is important in being able to achieve the goals and value you need in your testing. Ron Ulko from Domino’s said that their “ability to influence the Synack platform was a big differentiator in control.” Being able to leverage a crowdsourced security test in pre production is a game changer for DevSecOps integration and continuous security as a lifestyle.
  3. What does success look like in a crowdsourced security test? What sort of success metrics are there?
    Many companies believe finding fewer vulnerabilities equates to a stronger security posture. However, that mindset is erroneous. Measuring security through the quantity and severity of vulnerabilities fails to consider what a creative human may be able to accomplish when trying to break into a system. Additionally, with more companies moving to an Agile development style, new vulnerabilities are being introduced into an attack surface continuously. Quantity doesn’t cut it anymore. Each CISO agreed this question is important in understanding how a security vendor looks at the results they provide. Ron Ulko from Domino’s stated that it’s not just about fewer vulnerabilities, but it’s more about changing the game to help improve overall security. He likes to provide Synack data (which goes beyond count of vulnerabilities) to his development team so they are able to dissect it, and write more secure code to prevent vulnerabilities from appearing in the future. Synack’s Attacker Resistance Score provides a scoring system for how resistant assets are to attack. The longer continuous testing is implemented, the higher the number tends to get, with the goal being 100. This score is especially useful in understanding the progress of security testing and the overall improvement of an organization’s security posture. Vendors that provide intelligence such as location, severity, and timing of vulnerabilities found can help close the loop on development and remediation, which leads to stronger, continuous security across the organization.

CISO discussion at the Synack Suite - Black Hat 2019

It’s clear after this discussion that the questions a CISO once asked when evaluating a crowdsourced security vendor have changed. With the adoption of crowdsourced security testing becoming mainstream, technology that allows for customer control and visibility through the process is more important than ever. Synack is excited to keep fostering provocative discussions as the landscape continues to change. Until next year’s Black Hat!