The New AI-Enabled Offense Against Machine-Speed Adversaries

00:04 Well, howdy, folks. Thanks for joining this session. My name is Tim Nordvedt. I run the solutions architect team for North America here at Synack. We're an offensive security testing company. That basically means that I spend my day working with customers in real environments, real applications, helping them find risk before others do. But enough about me. Let me tell you about this session, what we're going to talk about, and kind of what this session isn't. So we're going to talk about three things AI-related. We're going to talk about how AI has changed the threat landscape, what AI has done to your risk as an organization,

00:38 and what you can do about it. What this isn't, this isn't a product pitch. I will, later on in this presentation, show you a little bit how we at Synack approach this near the end, but I'll let you know when we get to that point. Here's why I picked this topic and kind of wanted to use it for this session is over the last year, the conversation around AI and especially in the offensive security world has gotten very loud. There's been a lot of noise, and a lot of that noise is filled with vendor claims and the headline news, and neither of those are evidence. You can very easily get distracted by them,

01:10 but we're not talking hard facts when we look at those numbers. So everything I want to talk about in this session is I want to look past the headlines, look past the claims, and let's look at what the numbers actually say so we can see the current state of how AI is affecting the threat landscape. There are three resources in the chat available for you to download. One, I will reference throughout this talk. It's a survey we did with Omdia, the Omdia and Synack State of Agentic AI in Pentesting report. As I mentioned, I will reference some of those results in the talk today.

01:42 We also have a data sheet on Sara, our agentic AI pentesting solution. Sara stands for Synack Autonomous Red Agent. And lastly, we have a case study from one of our customers that operates in a regulated financial environment discussing how we can support them with reducing risk in that environment. So grab them whenever. Nothing in this talk is dependent on that, but they're great takeaway resources. So as we get into this conversation, I want to take you back, twenty-two years.

02:12 The reason I want to go back twenty-two years is I want to show you that we've been through similar inflection points. You know, when we start talking about AI, many say that AI is changing the game. While I agree with that, I actually like to say that AI is accelerating it rather than changing it. It may sound similar, but I feel there is a distinction. So let me explain. As I mentioned, let's go back twenty-two years to two thousand four when H. D. Moore open sourced Metasploit. For those that aren't familiar, Metasploit is an automated exploitation framework.

02:44 Basically, you log in to the Metasploit framework. You choose your exploit. You enter IP address, hit enter, and then Metasploit does all the work. Prior to when H. D. Moore open sourced Metasploit, running a professional-grade exploit, it took one of two things. It either took nation-state-level resources or took years of specialist skill. It was generally hard to access and utilize these exploits. And, honestly, that difficulty in itself was a security control for a lot of organizations because the inability to access the exploits did a lot of the defensive work for you.

03:20 But then along comes H. D. Moore, who open-sourced Metasploit. After that moment, a curious teenager with a laptop had access to those exploits now. Now I want to be precise about what changed here because this is going to frame the rest of the conversation and how I look at AI and how it's changing the threat landscape because in my opinion, the threat didn't change. The vulnerabilities that existed in your environment, in your application, the day after Metasploit was released were the exact same vulnerabilities that were there before. See, nobody invented a new class of attack.

03:53 There weren't new threat vectors that were created. What changed was the access curve to existing threats. The number of people who could reach those vulnerabilities and utilize them went up by orders of magnitude overnight. And that distinction matters because it tells you what you can do about it. See, if the threat changes, you need new controls. You need to change your stack in order to address these new attack vectors. But if the access curve collapses, you need more coverage faster. You need to figure out how to do what you're already doing just more efficiently.

04:24 Now bring that forward today. These are the headlines that we've all read on the right-hand side. Anthropic's Mythos model found thousands of previously unknown vulnerabilities across major operating systems and browsers with limited human steering. That's their headline. That's their characterization. And I want to put a pin in that for a minute because I'm going to come back to that and get more specific about it in about, you know, ninety seconds or so. But I also want to highlight that it wasn't just one lab. Anthropic, Mythos, they opened the door, but it was followed by others very soon after. OpenAI shipped GPT-5.5-Cyber in June.

04:58 Then on August eleventh, just very recently, they shipped GPT-5.6-Cyber, which is purpose built for vulnerability research and exploit development. The recent models completed ninety-five percent of advanced cybersecurity tasks, whereas the one in June completed just over fifty seven percent. That is a thirty eight point jump in eight weeks. Given when you're hearing this presentation, those stats have probably already changed. So the same pattern, the threat didn't change. What changed is the access curve to those threats collapsed again.

05:31 Now the obvious question is how much? And here's where I want to slow down, and I want to look a little closer at those Anthropic headlines and see what the numbers actually say. So before I get into these numbers, you know, I want to highlight that for decades, the scarce thing in our industry was finding vulnerabilities. That's why organizations invest in tools, and that's what you were buying when you bought a scanner. That's what you got when you opened a bug bounty program or you engaged your pentesting vendor or you created your internal red team and added headcount to your security team.

06:06 All these were aimed at one constraint: surfacing more of what is wrong, finding risk before someone else does. See, finding was hard. Finding was slow. Finding was expensive, but that constraint just ended. With these frontier models, finding is no longer the constraint. And the shape of how it ended is what matters. So let's look at these numbers. So Mythos flagged six thousand two hundred and two bugs at a high or critical severity. Of those, one thousand seven hundred and fifty two were sent to six separate security research firms, not Anthropic,

06:41 so genuine third parties, for review. Sixty-two point four percent of the reviewed bugs came back as confirmed genuinely high or critical, with a false-positive rate of nine point four percent, which is honestly not bad. It's in the industry range. The remainder of those were lowered in severity to mediums or lows. So of that one thousand seven hundred and fifty-two, verified high or critical, five hundred and thirty two were disclosed to maintainers, seventy five have been patched.

07:12 Now I pause there for a second because I want you to think about that number. Sixty two hundred vulnerabilities found, seventy five were fixed. That is one point two percent of what Mythos discovered actually got repaired. Now the easy conclusion is think, well, remediation is the new bottleneck. We've got a patching problem. Well, there is some truth to that. That is half right, but I actually want to focus on the other half, the half that's wrong because to me, that's the important half. So let's look back at these numbers and see where the drop off actually happens.

07:45 Of the findings that got independently reviewed, roughly a third were genuinely not severe. Five hundred and thirty made it as far as the maintainer, and seventy five got patched. Seventy five, it wasn't only seventy-five patched because patching is hard. They patched seventy-five because that is how many findings could actually be proved. That is how many findings someone could stand behind and validate, and invest resources to fix. So from where I sit, you know, the new constraint is not remediation. It's actually proving vulnerabilities.

08:17 So your remediation capacity is fixed. You've only got so many change windows. You've only got so many maintainers. You have a limited regression testing budget, and that number, you know, realistically does not move year over year. What determines your security outcome is whether you spend that capacity on the things that actually matter. See, I'll tie it back to a conversation I had a few months ago at a conference with the CISO of one of my customers that really started getting me thinking down this path. We were talking about, you know, our AI agentic offering. And the comment he made to me,

08:48 which really made me pause was he said, look, I already have more vulnerabilities than I can fix. Like, I've got a backlog of stuff that I just don't have the resources to fix. If you're going to tell me that AI is going to come in and just add to that list and find more vulnerabilities I can't fix, I'm not interested. What I need help with is understanding where I need to focus my remediation resources. I want to fix where the impact is and what's actually going to make a difference, and that's where I need help. And that's what I want to spend the rest of this session talking about is talking about the middle of that funnel.

09:21 How do we turn volume into a short list that you can then go and defend spending your change windows on remediation? And this is why, in my opinion, at least where we are in the current state, validation still requires humans. That's not a hedge. That's not a limitation. It's an architectural fact. And I threw a couple of stats on there from different reports I've read recently. One from Palo Alto Networks, they reported that, you know, they're utilizing some of the frontier models against their own products, and they're reporting roughly a thirty percent false-positive rate.

09:53 Cisco, in a recent release, their own specification put it pretty bluntly. Their phrase was a frontier model produces fluent, confident, plausible vulnerability claims that are wrong. That makes unreviewed output worthless. And Albert Ziegler, who runs AI at XBOW, puts it plainly that a model is a brain without a body. Now before we move on, I do want to address a conclusion that may come to mind as you're looking at this. Like, if you heard sixty two hundred vulnerabilities,

10:25 but only seventy-five got patched, you might start thinking, oh, this is just more hype. This is just, you know, a bunch of noise. But here's why this number should stop you, and I want you to pause for a minute. That same model, that same report about the sixty two hundred vulnerabilities where they dug into some Firefox vulnerabilities, and that model was able to turn that into a hundred and eighty one working attacks. The previous model managed to form two attacks. So it went from two to a hundred and eighty one attacks in one model. That is a capability jump. So finding is fast.

10:58 Exploiting is getting really fast. The slow part is fixing. And I'll be straight with you. I told you a minute ago that validation still needs humans. I firmly believe that today, and I believe that for the near future. But as capabilities evolve, if I were a betting man, it's probably not going to hold at scale as future AI capabilities evolve, but that is where we are today. So going back to that one point two percent, that isn't proof that, you know, the rest of these vulnerabilities are just noise and nothing happened.

11:30 What that is potentially creating is a gap that someone else, potentially, somebody you don't want to, gets to walk through. So what I talked about here, this is Anthropic's provability problem. I talked about proving being the constraint. So this is what they're dealing with. I want to turn the focus on you and your organization, what your problem is, because it's actually a bit worse, and let me show you why. See, Anthropic at least had all the findings in hand. They had everything, the entire landscape in front of them. The problem is just, you know, of that,

12:01 how do they prove what matters? While you have that same problem, you have three additional problems that I see stacked on top of it. So let me take them one at a time. First, discovery volume. And these numbers I'm going to talk about, just realize this is before we factored in AI and what the frontier models are doing. So the number forty eight thousand one hundred and eighty five CVEs that were published in two thousand twenty five, that is up over twenty percent from twenty twenty four, which was just under forty thousand. It's not a projection or an estimate.

12:31 That's the actual count. In fact, NIST had to restructure how they operate the National Vulnerability Database in April of this year, specifically because the intake of vulnerabilities is outpacing their process to manage them. So the pipe was already widening. This all took place before the frontier model showed up. So as we start to look at the twenty twenty six numbers when they come in, whatever those models add is on top of an already annual twenty percent increase. So the vulnerability discovery is massively increasing.

13:04 Second, and this is one I'm not sure everyone's fully absorbed yet as I'm out there talking to customers, is that the compliance clock has been compressed. CISA issued Binding Operational Directive 26-04 in June of this year. You know, not proposed, not for comments, but it's actually issued, and it's enforced for federal civilian agencies right now. It replaces a flat deadline with a five-tier risk model, and I want to be precise about that when you see the three days. I want to make sure you understand what that three-day requirement applies to.

13:34 It is when a vulnerability is in the Known Exploited Vulnerabilities catalog, and it yields total system control and is fully automated. So there's multiple conditions that have to be met, in order for that three-day requirement to apply. So most KEV-listed items get fourteen days; lower risk combinations get sixty. There's even a low tier where it's basically just fix it at your next upgrade. Now I understand that most people watching this session, they don't work for a federal civilian agency, but here's why I bring this up and why it matters.

14:06 CISA explicitly encourages private-sector adoption. And in my experience, the practical effect of directives like this is that within about twelve to eighteen months, auditors, your cyber insurers, your large customers, they are going to start to use this as a reference for what reasonable looks like. So whether this applies to you or not, it is something that is going to come on most of our radars in the near future. And the third problem that compounds this provability issue is one that you can actually measure in your program today,

14:39 and that is your assessment cadence and coverage. I mentioned in the chat we have one resource, the Omdia and Synack research report, where we interviewed security leaders. One of the questions we asked them is: of your entire attack surface, what are you actually testing on an annual basis? And the answer actually surprised us because the average organization only tests thirty two percent of their attack surface in a given year. One in three organizations that responded actually test twenty percent or less,

15:10 which brings me to this number on the left-hand side, sixty eight percent. That's the share of the average attack surface that goes untested in a year. Not under tested, not tested late, but not even looked at. So let's put all those three issues together. Discovery volume is rising twenty percent every year before AI enters the picture. A compliance clock that at its most severe tier gives you seventy-two hours to fix an issue. And two-thirds of your environment has never even been examined.

15:40 And then here's the uncomfortable structural fact underneath all of these: you can't hire your way out of this gap. The math just doesn't work. The volume is growing faster than any team you could realistically staff. And even if you could, the talent just isn't there even if budgets were, which means at some point, you are going to have to buy or build something in order to address these problems. And I can certainly take a guess that the words AI-powered are going to be in that description. But what does that actually mean when we talk about AI-powered? So let's dig in.

16:17 So this is where I need to stand on my soapbox for just a minute. If you can just bear with me because AI-powered has been doing a tremendous amount of work in the security market in the recent past. Because here's the thing, AI is not new. AI has been around since the nineteen fifties. Alan Turing was asking, can machines think, before I dare say most of us were even born. So AI, artificial intelligence, we have all been using it for decades; it just has not been termed AI.

16:49 So there's a lot of variations to AI, and it's a pretty broad term. But I am going to simplify and I want to put it down to three different tiers. And these aren't just variations on a theme. These are three different uses of AI, with different engineering, different costs, radically different capabilities. Tier one is automation. This is the AI that we've all been using for decades. It's high-speed pattern matching of known signatures. It's what much of the industry has been shipping for decades. And to be fair, it works. It's cheap. It's fast.

17:20 It scales beautifully. But what it can't do is reason about a novel attack path. It finds things where there's already a known signature. It is an older capability that's been given a fresh coat of paint. Tier two is generative AI wrapped around a workflow. It is the ChatGPTs. It is the Claudes. It is what most of us have been using for the past few months, the past year or so: better reports, faster summaries. It's a cleaner interface, natural language querying of your findings,

17:50 and it is valuable. I do not want to be dismissive of it because report quality is a real problem, and bad reports do come with a real cost. But let us be clear about what it improves. It improves how you consume the output. It does not actually improve the output. If the underlying engine is a tier-one scanner, tier two gives you a better interface on an old scanner, but the findings are identical. Tier three is agentic, and this is where we need to focus. This is the system that can plan. It's the system that can pivot when something doesn't work.

18:23 It can observe results and take actions accordingly. It can chain issues together. It is categorically different from pattern matching. This is the tier that Mythos and other frontier models, live in. And honestly, this is the tier Synack lives in with our own system we're developing. This tier three is the Metasploit level shift. Everything in the first two tiers is an efficiency gain. The third tier is a genuine capability change. So we know we need to be in tier three:

18:54 real agentic reasoning. This is table stakes. This is necessary. But, you know, that alone is nowhere near sufficient. And in the next slide here, I'm going to talk about two AI labs, and how their recent published reports explain why that is. And the reason I want to discuss this, you know, as the title says, it's the harness, not the model. When I, as a solutions architect, when I'm sitting down talking to customers and they're evaluating our AI pentesting solution, ninety nine percent of the time, the first question I'm asked is what model are you using?

19:25 Do you have access to Mythos? Which frontier models are part of your solution? And I understand the instinct because that's the part you can look up. That's the part that's easy for most people to grasp. Now I mentioned Synack. We're in the offensive security testing world. So I want you to think about the analogy of how you would evaluate a pentest firm. If you're evaluating a pentest firm for your security needs, your first question is not going to be, tell me who your new hires were. Tell me about your onboarding process for pentesters. You're going to ask what's your methodology.

19:57 Tell me about your overall engagement experience. What are your rules of engagement? How do you enforce those? What happens when something goes wrong? See, the model is the new hire. The harness is the overall engagement and process. But everybody's so focused on buying the new hire, not the holistic engagement. Now I mentioned I want to reference two reports that talk about why the harness should be the focus and not the model. First one is Anthropic. They ran a multi-agent setup where Claude Opus four was the lead agent with Claude Sonnet four as subagents underneath of it,

20:30 and they compared that to a single instance of Opus four working alone. The orchestrated version outperformed the single agent by ninety-two percent on their internal research evaluation. Now two caveats I want to point out. One, this is a research evaluation. It's not a cybersecurity benchmark. So the task was finding and synthesizing generic data not specific to vulnerabilities. The other one, and I think this is the more important one to pay attention to, is that the orchestrated multi-agent approach consumed roughly fifteen times the credit cost of the single model.

21:03 We all want performance. That's where we need to be, but we need to understand performance comes at a cost. Now to the second example I want to pull from, and this one sits squarely in our security domain. In May, Microsoft published a system they called MDASH, a multi-model agentic scanning harness. More than a hundred specialized agents running across a mix of frontier and smaller distilled models, and they ran it against CyberGym, which is over fifteen hundred real-world vulnerability tasks. They scored eighty eight point four percent,

21:34 which is the top score on the public leaderboard. The next best score was eighty three point one percent. Now I want you to take note of who scored that eighty three percent because the entry that Microsoft beat was Mythos. The hundred orchestrated agents beat the frontier model, not a different model, not a better model, just the same class of models deployed differently. So the harness, not the model, the ability to orchestrate, decompose a task, validation between steps, that is where our focus needs to be. Now I want to name something out loud because I don't want you to feel like I'm having it both ways.

22:09 I just cited a benchmark to make a point. But what I'm about to say is to tell you that benchmarks actually don't transfer to real-world scenarios. But both of these statements can be true. Here's why. See, when it comes to why they don't transfer to real-world scenarios, MDASH ran CyberGym in its level one configuration. What that means is that level one hands the testing system the entire vulnerable source code and a high-level description of the vulnerability you're looking for.

22:40 So read that result again with that thought process in mind that Microsoft needed more than a hundred orchestrated agents across an ensemble of frontier and distilled models to win a benchmark that gives you source code and tells you what you're looking for. Now picture the real-world target where you don't get anything. This is why benchmarks are valid, but also limited at the same time because when you start talking real-world, real-world environments are messy. Real authentication has to be navigated, not bypassed.

23:12 There are session timeouts. There's legacy fingerprints on a host system that nobody's inventoried for years. There's that back end that three acquisitions ago, somebody bolted on a system and the person who did that left in twenty nineteen. See, there is no source code. There is no description of a bug. There's no guarantee that there's even a bug at all. And it's that gap between a benchmark that hands you the answer key and the environment that hands you nothing. That's the single biggest reason that capability demos don't survive contact with production.

23:43 So I'll dig into a little more of that here in a couple slides, but it's why the right mental model coming into this isn't just we bought and deployed a tool. It's thinking about the way you onboard an agent team the same way you would onboard a human team. You know, in the world of offensive security, you scope it, you give it credentials, give it rules, you watch how it does on a target, you build understanding and trust before you trust it with one that is actually production and could have a bigger impact. And you build a feedback loop for when things go wrong.

24:13 See, in the real world, you would never put a human team on a production target without rules of engagement. Same goes for agents, except now the enforcement has to happen at machine speed. So after the model question, the next question I always get asked when it comes to agentic AI solutions is guardrails. So here is what guardrails actually look like in production. On the left is the shape of the answer. It's the policy layer that lives outside the model. Concretely, you know,

24:44 a runtime classifier that evaluates every action an agent is about to take against a specific customer's rules of engagement. It's across a swarm of agents rather than one at a time. And this is critical. It's the ability to stop an action mid-execution, not flag it, but stop it. Plus it has to have the ability to do the boring parts that really matter in practice. It's the scope boundaries, the rate limits, the target restrictions. They have to be held externally where a model doesn't have the ability to negotiate around them. And you need a full audit trail of what ran and why.

25:18 Because the first question after an incident is always what happened. And the answer of we're not sure, that is not an acceptable answer. And I have to be honest about where this came from because we here at Synack, we did not design all this upfront. It got built after our system left the lab and met real-world customer environments. We made a bunch of assumptions when we made this transition from the lab to production, and we found out which assumptions held true and which ones didn't. And this is how we develop a solution that produces results in real-world scenarios.

25:50 Now, on the right side, why prompts aren't enough: it's because a prompt is not a control. And several reasons why prompts, they can be reasoned around. An instruction inside a context window is an input to a model's reasoning. It's not a constraint. It is a strong suggestion competing with everything else in that context. Also, agents' tasks decompose over time. Every task splits into a subtask. You get another opportunity for the original intent to get slightly lost. It's the old analogy of the telephone game.

26:21 When you have one agent telling an agent telling another agent, things can change over time and drift. When you look at a swarm approach, a swarm can amplify one bad inference. The part that I think a lot of people underestimate is that parallelism is exactly what makes agent systems so useful and so powerful, but it's also what turns one bad judgment into fifty simultaneous bad actions. Lastly, rules of engagement are contractual, not advisory. You signed something.

26:51 The model didn't, and that asymmetry is a problem. Post hoc detection is too late. Detection is a fine strategy, but when you have minutes or even seconds at machine speed, finding out afterward is too late. So here's the thing. You are going to deploy agents. Maybe offensive ones, probably defensive ones. Almost certainly, you're already deploying operational ones that touch production systems, and they may be security-related or have nothing at all to do with security.

27:23 But every one of them has the same structural problem. It's a capable nondeterministic system taking actions faster than humans can review them. So whatever agents you deploy or whatever agents you purchase and deploy, you know, offensive, defensive or operational, they need a policy layer that lives outside the model. And, you know, a control plane, what we're talking about answers one question. Can I stop it? There's another question we need to ask, and this goes back to provability being the new constraint is, should I believe it?

27:54 And your peers have already answered and put a number to what believable looks like. So this slide here, this is peer data. This is not a Synack specification. This is actually output from the report that we attached in the chat that you can download. But this is data from two hundred security leaders that Omdia reached out to, and it tells us what the market has already decided is table stakes. So let's start with the distribution on the left-hand side. When asked what accuracy an AI-driven testing system would need to hit before they trusted it alongside manual testing,

28:28 the answer clusters hard in the eighty five to ninety four percent band. Sixty nine percent of respondents put the floor at eighty-five percent or above, and almost nobody accepted below seventy five percent. Now hold that against the numbers that I gave you earlier with a nine point four percent false-positive rate from Mythos and Palo Alto reporting roughly thirty percent from the frontier models across all their products, and then sixty two point four percent of reviewed findings were actually confirmed as genuinely severe when we look at that Mythos down funnel.

29:00 And what this points out is there's a gap between reality and what the market demands. And that gap between the two is precisely where a lot of AI security tooling is going to fail when it comes to their first review. Now the other three figures on the right-hand side, these are about trust rather than accuracy. And they're the ones that if you sit on the governance side of the summit, you should pay attention to. Ninety-three percent say guardrails are critical or important. That is near unanimous. And it's actually interesting that people care more about bounded behavior than about being right.

29:37 Fifty eight percent required transparent AI decision-making, meaning show me why. A finding that you can't explain is a finding you can't act on, and, actually, it's a finding that you can't defend to a regulator. Fifty-four percent require industry certifications and standards. Here is why I present this slide and how you can use it. This is the bar to hold your own tooling to, whether you build it or buy it. If you have an AI-driven solution deployed in production right now and you can't tell me the false-positive rate,

30:07 then you honestly don't know whether you're above or below this line your own peers just drew. So we've got eighty five percent accuracy as a standard. We've got guardrails, transparency, certifications. This is a demanding bar, and it is what your peers have set as table stakes. And the fair question is whether anybody actually, you know, delivers this in production. So let me show you our attempt here, and I want to be specific about where humans are in this overall equation.

30:37 So we are twenty-plus minutes into this presentation, and this is the first time I'm really going to talk about what we do at Synack, and that is intentional and deliberate. My goal is not to come and just kind of tell you what we're doing at Synack. I want everything to be able to stand on its own. So whether or not you speak to us after this presentation, I want you to have knowledge that you can apply to the agents you are looking at buying or building. So when it comes to Synack, three numbers I want to touch on first. First, fifteen hundred-plus vetted security researchers on our Synack Red Team bench. This is not an open sign-up form.

31:11 This is a vetted, background-checked, continuously monitored and assessed community. So second one, thirteen years in market, which means thirteen years of exposure to real-world production environments, messy authentication, legacy hosts, the acquisitions nobody documented. We've been navigating these issues for well over a decade. Lastly, a private, controlled model, not an open bug bounty, but known researchers, defined rules of engagement, full traffic attribution.

31:42 Now let me connect those back to this presentation, why I put this slide here rather than the front of the deck. Earlier, I mentioned that you should onboard an AI agentic team the same way you onboard a human pentest team. This is what we've benchmarked our agentic solution against. Fifteen hundred researchers working on real targets for thirteen years. That gives us two things that you just can't buy quickly. First, a record of what real findings look like versus a false positive across thousands of live engagements.

32:14 Second, people who can check the AI's work against it, against reality when something goes wrong. For a competitor who started twelve to eighteen months ago, this is a real-world problem because you just cannot raise capital and license the same models and expect to have the same output when we've been doing this for thirteen years. So moving on to what we've delivered, we have a division of labor. See, we believe that we take a symbiotic approach, with AI and humans, rather than full autonomy alone,

32:44 but humans alone will not address the problem either. So there's a division of labor. You know, on the AI side, you know, I mentioned our solution, Sara, Synack Autonomous Red Agent, handles the recon, the asset enumeration, at machine speed. It can handle that tier one stuff we talked about, the known pattern detection. It can also handle novel exploits, trying to find paths with privilege escalations, continuous coverage, you know, autonomous triage and deduplication, which is where most of the false-positive burden actually gets absorbed.

33:17 But we still have humans in the process because as powerful as AI is today, as capable as these frontier models are, AI alone is not adequate. The latest models, there are still certain limitations of things they can't do. See, humans can handle horizontal chaining across disconnected systems. They can understand business logic and the trust relationship attacks that AI is just not there yet. Humans have the ability to apply judgment to ambiguous findings.

33:49 And I just want to be clear that the right-hand column of humans is a design choice. It's not a limitation we're apologizing for. It's because, as I mentioned, there are certain categories that humans are just generally better at today, and pretending otherwise would fail that accuracy standard we talked about on the previous slide. And this split isn't just, you know, our opinion. You know, you can read this in the Omdia survey results, but sixty four percent of leaders prefer an agent-led model with human oversight. And they're actually one point six times more likely to describe oversight as a permanent,

34:21 not a temporary control once deployed, which tells you that the people closest to this are less confident about full autonomy, not more. So that's how we build it. Now I know you especially didn't come here to hear about architecture. You came here for something that you can use in your next conversation. So here's how you can take this and apply it to your conversations after this session. Five things. This can apply to whether you're building agents yourself or you're going to buy them from a vendor.

34:52 The first three on the left-hand side, these are controls that you put in place. The second two on the right-hand side, these are claims to verify before you sign anything. Number one, a validation gate. Nothing reaches a queue unreviewed. And I want to be clear that while I just said that we believe in the AI and human combination, that doesn't always mean it has to be a human review, human validation before it gets to the queue. It is human-in-the-loop where the outcome matters. We don't want to lose the advantage of machine speed by putting human stop gates at every process for the agentic

35:28 AI. Number two, a runtime policy layer that can stop an agent mid-action. That's the control plane that we described a few slides ago. It's whatever you deploy, ask where the enforcement lives. Does it live inside the prompt or outside the model? Number three, and this is specifically for, you know, the offensive agents that are deployed. Found and confirmed exploitable have to be two separate steps with two separate agents. A finding says this looks vulnerable.

35:59 Confirmed exploitability means we made it work and we can show you proof. Here's the request that succeeded, the data that came back, the account that was taken over. If one system is doing both, that's like grading your own homework. And since severity is a driver in your remediation queue, if you're fixing in the order that the model felt confident about, that is not as ideal as fixing in the order that is actually going to reduce risk in your environment. So number four, as we move to the right-hand side: orchestration. You know, not a single model.

36:29 It's what's actually around the model. Is there a decomposition of the task? Are there retry processes? Is there cross checking? You know? And in reality, you also need to ask, what is the token cost? Because, as we mentioned, Anthropic's own published figure is that when you look at this orchestration approach versus single model approach, their figures were roughly fifteen times the cost. And last, number five, benchmark literacy. While benchmarks have value, you also need to understand their limitations.

37:00 So when you hear that benchmark score, ask the context around that. Were they given source code? Were they given a description of a bug? Was there a guarantee that bugs even exist at all? Is it that CyberGym level-one problem we discussed? It's understanding the difference between an impressive number and a relevant one. So ask your engineers or any vendor you're talking to these same five questions. These answers are going to tell you where the harness ends and where the hype begins.

37:30 And I want to be straightforward with you. I think we at Synack answer these questions well, but I've deliberately not framed these questions as a reason to pick us because I wanted a version of this that's useful to you, a framework that you can apply across the board because that's the one you can take away. If you ask us these five questions, and you do not like the answer, that is a legitimate result. You should act on it, which brings me back to where we started. Let's go back to two thousand four Metasploit. The organizations that came out of the transition ahead were not the ones that, you know,

38:04 increased their perimeter defenses. They weren't the ones that could react the fastest to every new exploit that was announced. They were the ones that looked at their program and decided to evolve and reshape what they were doing. They stopped treating security testing as an event that happens once or twice a year, and they started looking at how can we do this on a continuous basis. So look at today: Mythos, GPT-5.6-Cyber, MDASH. You know, we're at the same inflection point. The access curve has collapsed again,

38:34 and the question is not whether to respond, it's whether you respond before your adversaries. And one practical note about the environment that we're making these decisions in, a recent report from Forrester projects that enterprises will defer about a quarter of their planned AI spend to twenty twenty seven, as CFOs start asking for the ROI evidence that most teams aren't able to produce yet. And note that deferred does not necessarily mean canceled. It is just pushed off. What this means is that the AI security investment that you're looking to propose in the next two quarters is going to face a much harder business case and validation than the same

39:10 proposal would have faced a year ago. So make sure you bring the numbers as you propose a solution. It is why I dug so much into the evidence today rather than the hype, so you have the ammunition. If you want to prove this works in a regulated environment, as I mentioned, we attached the Iberia Cards case study in the chat. They are in financial services, under a European regulatory regime. It is a very useful read on governance and how agentic AI operates in practice. So let me close with one question I'd like you to be able to answer.

39:43 How much of what you found last year could you actually prove? Not findings you generated, not the outputs of the scanners, but how much could you actually stand behind well enough that you're willing to spend a change window on them? That number, not the number you found, is what your program actually delivered. Everything else was volume that you just paid to produce. So let us just realize, finding stopped being the hard part. Proving is where we need to focus our efforts now. If you would like to learn more about that,

40:15 we would love for you to reach out to us so we can continue that conversation. But again, thanks for taking the time to sit through the session and enjoy the rest of the webinar.

No lines match that search.