The New AI-Enabled Offense Against Machine-Speed Adversaries

00:04 Well, howdy, folks. Thanks for joining this session. My name is Tim Nordvett. I run the solutions architect team for North America here at SYNNAC. We're an offensive security testing company. Basically means that I spend my day working with customers in real environments, real applications, helping them find risk before others do. But, enough about me. Let me tell you about this session, what we're gonna talk about, and kind of what this session isn't. So we're gonna talk about three things AI related. We're gonna talk about how AI has changed the threat landscape, what AI has done to your risk as an organization,

00:38 and what you can do about it. What this isn't, this isn't a product pitch. I will, later on in this presentation, show you a little bit how we at SYNNAC approach this near the end, but I'll let you know when we get to that point. Here's why I picked this topic and kinda wanted to use it for this session is over the last year, the conversation around AI and especially in the offensive security world has gotten very loud. There's been a lot of noise, and a lot of that noise is filled with vendor claims and the headline news, and neither of those are evidence. They can very easily get distracted by them,

01:10 but we're we're not talking how hard facts when we look at those numbers. So everything I wanna talk about in this session is I wanna look past the headlines, look past the claims, and let's look what the numbers actually say so we can see where the current state of how AI is affecting the threat landscape. There's three resources, in the chat available for you to download. One, I will reference throughout this talk. It's a it's a survey we did with Omnia, the Omnia and Synnex data of Agenetic AI pen test report. As I mentioned, I will reference some of those results in the talk today.

01:42 We also have a data sheet on SARA, our AgenTik AI pen testing solution. SARA stands for Synact Autonomous Red Agent. And lastly, we have a case study from one of our customers that operates in a regulated financial environment discussing how we can, support them with reducing risk in that, environment. So grab them whenever. Nothing in this talk is dependent on that, but they're great takeaway resources. So as we get into this conversation, I wanna take you back, twenty two years.

02:12 The reason I wanna go back twenty two years is I wanna show you that we've been through similar inflection points. You know, when we start talking about AI, many say that AI is changing the game. While I agree with that, I actually like to say that AI is accelerating it rather than changing it. It may sound similar, but I feel there is a distinction. So so let me explain. As I mentioned, let's go back twenty two years to two thousand four when HD Moore open sourced Metasploit. For those that aren't familiar, Metasploit is an automated exploitation framework.

02:44 Basically, you log in to the Metasploit framework. You choose your exploit. You enter IP address, hit enter, and then Metasploit does all the work. Prior to when H. C. Moore open sourced Metasploit, running a professional grade exploit, it it took one of two things. It either took nation state level resources or took years of specialist skill. It was generally hard to to access and utilize these exploits. And, honestly, that difficulty in itself was a security control for a lot of

03:14 organizations because the inability to access the exploits did a lot of the defensive work for you. But then comes along HD Moore, open source Metasploit. After that moment, a curious teenager with a laptop had access to those exploits now. Now I wanna be precise about what changed here because this is gonna frame the rest of the conversation and how I look at AI and how it's changing the threat landscape because in my opinion, the threat didn't change. The the vulnerabilities that existed in your environment, in your application,

03:45 the day after Metasploit were released were the exact same vulnerabilities that were there before. See, nobody invented a new class of attack. There weren't new threat vectors that were created. What changed was the access curve to existing threats. The number of people who could reach those vulnerabilities and utilize them went up by orders of magnitude overnight. And that distinction matters because it tells you what you can do about it. See, if the threat changes, you you need new controls. You need to change your stack in order to address these new

04:15 attack vectors. But if the axis curve collapse, you need more coverage faster. You you need to figure out how to do what you're already doing just more efficiently. Now bring that forward today. The these are the headlines that we've all read on the right hand side. Anthropics, Mythos model found thousands of previously unknown vulnerabilities across major operating systems and browsers with limited human steering. That that's their headline. That's their characterization. And I'm I wanna put a pin in that for a minute because I'm gonna come back to and get more specific about that in about, you know, ninety seconds or so.

04:46 But, I also wanna highlight that it wasn't just one lab. Anthropic, Mythos, they opened the door, but it was followed by others very soon after. OpenAI shipped g t GPT five point five cyber in June. Then on August eleventh, just very recently, they shipped, five point six cyber, which is purpose built for vulnerability research and exploit development. The recent models, complete in ninety five percent of advanced cybersecurity task, where the one in June completed just over fifty seven percent. That is a thirty eight point jump in eight weeks.

05:19 Given and you're hearing this presentation, those stats have probably already changed. So the same pattern, the threat didn't change. What changed is the access curve to those collapse to those threats that collapsed again. Now the obvious question is how much? And here's where I wanna slow down, and I wanna look a little closer at those anthropic headlines and see what the numbers actually say. So before I get into these numbers, you know, I wanna highlight that for decades, the scarce thing in our industry was finding

05:49 vulnerabilities. That's why in the organizations, they invest in tool, and that's what you were buying when you bought a scanner. That's what you got with when you opened a bug bounty program or you engaged your pen test and vendor or you created your internal red team and added headcount to your security team. All these were aimed at one constraints, surface and more of what's wrong, find and risk before someone else does. See, find and was hard. Find and was slow. Find and was expensive, but that constraints just ended.

06:19 With these frontier models, it it's finding is no longer the constraint. And the shape of how it ended is what's mattered. So so let's look at these numbers. So mythos flagged six thousand two hundred and two bugs at a high or critical severity. Of those, one thousand seven hundred and fifty two were sent to six separate security research from non anthropic zone people with genuine third parties, for review. Sixty two point four percent of the review bugs came back as confirmed generally higher critical false positive rate of

06:51 nine point four percent, which is honestly not bad. It's in the industry range. The remainder of those were lowered in severity to mediums or lows. So of that seven hundred and fifty two d, you know, verified higher critical, five hundred and thirty two were disclosed to maintainers, seventy five have been patched. Now I pause there for a second because I I want you to think about that number. Sixty two hundred vulnerabilities found, seventy five were fixed. That is one point two percent of what Mythos discovered

07:24 actually got repaired. Now the easy conclusion is think, well, remediation is the new bottleneck. We we've got a patch in problem. Well, there is some truth to that. That is half right, but I actually wanna focus on the other half, the half that's wrong because to me, that's the important half. So let's look back at these numbers and see where the drop off actually happens. Of the findings that that got independently reviewed, roughly a third were generally not severe. Five hundred and thirty made it as far as the maintainer,

07:54 and seventy five got patched. Seventy five, they it wasn't only seventy five patched because patching is hard. They patched seventy five because that's how many findings that could actually be proved. That's how many findings could actually someone could stand behind and validate, invest in resources to fix. So from where I sit, you know, the new constraint is not remediation. It's actually it's actually proven vulnerabilities. So your remediation capacity is fixed. You've only got so many change windows. You've only got so many maintainers. You have a limited regression test and budget,

08:26 and that number, you know, realistically does not move year over year. What determines your security outcome is whether you spend the capacity on the things that actually mattered. See, I'll it back to a conversation I had a few months ago at a conference with the CISO of one of my customers that really started getting me thinking down this path is that we were talking about, you know, our AI agentic offering. And the comment he made to me, which really made me pause was he said, look, I I already have more vulnerabilities that I can fix. Like, I've got a backlog of stuff that I just don't have the

08:56 resources to fix. If you're gonna tell me that AI is gonna come in and just add to that list and find more vulnerabilities I can't fix, I'm not interested. What I need help is understanding where I need to focus my remediation resources. I I wanna fix where the impact is and what's actually gonna make a difference, and that's where I need help. And that's what I wanna spend the rest of this session talking about is talking about the middle of that funnel. How do we turn volume into a short list that you can then go and defend spending your change windows on remediation.

09:29 And this is why, in my opinion, at least where we are in the current state, validation still requires human. That's not a hedge. That's not a limitation. It's an architectural fact. And threw a couple stats on there from different reports I've read recently. One from Palo Alto Networks, they they reported that, you know, they're utilizing some of the frontier models against their own products, and they're reporting roughly a thirty percent false positive rate. Cisco, in a recent recent release, they their own specification put it pretty bluntly where their phrase was a frontier model produces fluent,

10:01 confident, plausible vulnerability claims that are wrong that makes a that makes unreviewed output worthless. And Albert Ziegler, who runs AI, a Expo, puts it plainly that a model without a brain is just a body. Now before we move on, I I do want to kind of kill a conclusion that maybe for a minute as you're looking at this. Like, if you heard sixty two hundred vulnerabilities, but only seventy got patched, you might start thinking, oh, this is this is just more hype. This is just, you know, a a bunch of noise.

10:31 But here's why this number should stop you, and I want you to pause for a minute. That same model, that same report about the sixty two hundred vulnerabilities where they dug into some Firefox vulnerabilities, and and that model was able to turn that into a hundred and eighty one working attacks. The previous model managed to form two attacks. So it went from two to a hundred and eighty one attacks in one model. That is a capability jump. So finding is fast. Exploiting is getting really fast.

11:01 The slow part is fixing. And I'll be straight with you. I told you a minute ago that validation still needs humans. I firmly believe that today, and and I believe that for the near future. But as capabilities evolved, if I were a betting man, it's probably not gonna hold in scale for, know, as the future, AI capabilities evolve, but that's where we are today. So going back to that one point two percent, that isn't proof that, you know, the rest of these vulnerabilities are just noise and nothing happened. What that is is potentially creating a gap that someone

11:33 else, potentially, somebody you don't want to gets to walk through. So what I talked about here, this is Anthropic's provability problem. I talked about proven is the constraint. So this is what they're dealing with. I wanna turn the focus on on you and your organization, what your problem is, because it's actually a bit worse, and and let me show you why. See, Anthropic at least had all the findings in hand. They they had everything, the entire landscape in front of them. The problem is just, you know, of that, how do they prove what matters?

12:03 While you have that same problem, you have three additional problems that I see stacked stacked on top of it. So so let me take them one at a time. First, discovery volume. And these numbers I'm gonna talk about, just realize this is before we factored in AI and what the frontier models are doing. So the number forty eight thousand one hundred and eighty five CVEs that were published in two thousand twenty five, that it's that is up over twenty percent from twenty twenty four, which was just under forty thousand. It's not a projection or an estimate. That that's the actual count.

12:33 In fact, NIST had to restructure how they operate the national vulnerability database in April of this year, specifically because the intake of vulnerabilities is outpacing their process to manage them. So the pipe was already widening. This all took place before the frontier model showed up. So as we start to look at the twenty twenty six numbers when they come in, whatever those models add is on top of an already annual twenty percent increase. So the the vulnerability discovery is massively increasing.

13:04 Second, and this is one I'm not sure everyone's fully absorbed yet as I'm out there talking to customers, is that the compliance clock clock's been compressed. CISA, they issued binding operational directive twenty six dash o four, in June of this year. You know, not proposed, not for comments, but it's actually issued, and it's enforced for federal civilian agencies right now. It replaces a flat deadline with a five tier risk model, and I wanna be precise about that when you see the three days. I wanna make sure you understand what that three day requirement applies to.

13:34 It's when a vulnerability is in the known exploitable vulnerability category, and it yields total system control and is fully automated. So there's multiple conditions that have to be met, in order for that three day requirement to apply. So most Kev listed items, they get fourteen days, lower risk combinations get sixty. There's even a a low tier where it's basically just fix it extra next upgrade. Now I understand that most people watching this, session, they don't work for federal civilian agency, but here's why I bring this up and why it matters.

14:06 Assisted CISA explicitly encourages private sector adoption. And in my experience, the the practical effect of directives like this is that within about twelve to eighteen months, auditors, your cyber insurances, your your your large customers, they're gonna start to use this as a reference as for what reasonable looks like. So whether this applies to you or not, it is something that is gonna come on most of our radars in your near future. And the third problem that that it's it kinda compounds this

14:36 provability issue is one that you can actually measure in your program today, and that is your assessment cadence and coverage. I mentioned in the chat, we have the one resources, the Omnia and Synak research report where we interviewed security leaders. One of the questions we asked them is what is the of your entire tax surface, what are you actually testing on an annual basis? And the answer actually surprised us because the average organization only tests thirty two percent of their attack surface in a given year. One in three organizations that responded actually test less

15:08 twenty percent or less, which brings me to this number on the left hand side, sixty eight percent. That's the share of the average attack surface that goes untested in a year. Not under tested, not not tested late, but not even looked at. So let's put all those three issues together. Discover volume rise in twenty percent every year before AI enters the picture. A compliance clock that at its most severe tier gives you seventy two hours to fix a an issue. And two third of your environment has never even been examined.

15:40 And then here's the, uncomfortable structural fact underneath all of these is that you can't hire your way out of this gap. The math just doesn't work. The volume is growing faster than any team you could realistically staff. And even if you could, the talent just isn't there even if budgets were, which means at some point, are going to have to buy or build something in order to address these problems. And, it's most certainly I I I can take a gander that the words AI powered are gonna be in that description.

16:10 But what does that actually mean when when we talk about AI powered? So so let's dig in. So this is where I I need to stand on my soapbox for just a minute. If you can just bear with me because AI powered has been doing a tremendous amount of work in the security market in in the the recent past. Because here's the thing, AI is not new. AI has been around since the nineteen fifties. Alan Turn has been asking, can machines think before I I bet to say most of us were even born.

16:40 So AI, artificial intelligence has we've all been using it for decades, just not it hasn't been termed AI. So there's a lot of variations to AI, and it's a pretty broad term. But I I I am gonna simplify and I wanna put it down to three different tiers. And and these aren't just variations on a theme. These are three different uses of AI, three different engineering, different, you know, cost, radically different capabilities. Tier one is automation. This is the AI that we've all been using for decades.

17:11 It's high speed pattern matching of known signatures. It's what much of the industry has been shipping for for decades. And to be fair, it it works. It's cheap. It's fast. It scales beautifully. But what it can't do is reason about a novel attack path. It it finds things where there's already a known signature. It is an older capability that's been given a fresh coat of paint. Tier two is generative AI wrapped around a workflow. It's the chat GBTs. It's the the clogs. It's what we what most of us kind of been using for

17:41 for, past few months, past year or so is better reports, faster summaries. It's a cleaner interface, natural language querying of your findings, and it's it's valuable because I don't wanna be dismissive it because but report quality, is a real problem, and bad reports do come with a real cost. But let's be true. Let's be clear about what it improves. It improves how you consume the output. It it does not actually improve the output. If the underlying engine is tier one scanner, tier two gives you a better interface on an old scan in

18:13 scanner, but the findings are identical. Tier three is agentic, and this is where we need to focus. This is the system to plan. It's it's the system that can pivot when something doesn't work. It can observe results and take actions accordingly. It can chain issues together. It can it's category it's categorically different from pattern matching. This is the tier that mythos and other frontier models, live in. And, honestly, this is where SYNACK this is the tier we live in with our own system we're developing. This tier three is the metasploit level shift.

18:45 Everything in the first two is, two first two tiers is efficiency gain. The third tier is a genuine capability change. So we know we need to be in tier three real agentic reason. This is table stakes. This is necessary. But, you know, that alone is nowhere near sufficient. And in the next slide here, I'm gonna talk about two AI labs, how how they published recent reports of why that is, recently. And the reason I wanna discuss this, you know, as the title says, it's the harness, not the model. When when I, as a solutions architect,

19:16 when I'm sitting down talking to customers and they're evaluating our AI pen testing solution, ninety nine percent of the time, the first question I'm asked is what model are you using? Do you have access to mythos? Which frontier models are are are part of your solution? And I understand the instinct because that that's the part you can look up. That's the part that's easy for most people to grasp. Now I I mentioned SYNNAC. We're we're in the offensive security testing world. So I want you to think about the analogy of how you would evaluate a pen test firm. If you're evaluating a pen test firm for your for your security

19:47 needs, your first question is not gonna be, tell me who your new hires were. Tell me about your your onboarding process for pen testers. You're you're gonna ask what's your methodology. Tell me about your overall engagement experience. How what are your rules of engagement? How do you enforce those? What happens when something goes wrong? See, the model is the new hire. The harness is the overall engagement and process. But everybody's so focused on buying the new hire, not the holistic engagement. Now I I mentioned I want to reference two reports that talk

20:19 about why the harness should be the focus and not the model. First one is Anthropic. They they ran a multi agent setup where Claude Opus four was the lead agent with Claude Sonnet four as sub agents underneath of it, and they compared that to a single instance of Opus four working alone. The orchestrated version outperformed the single agent by ninety two percent on their internal research valuation. Now two caveats I wanna point out. One, this is a research evaluation. It's not a cybersecurity benchmark. So the task was finding and synthesizing generic data not

20:51 specific to vulnerabilities. The other one, and I think this is the more important one to pay attention to, is that the orchestrated multi agent approach consumed roughly fifteen times the credit cost of the single model. We all want performance. That's where we need to be, but we need to understand performance comes at a cost. Now to the second, example I wanna pull from, and this one sits squarely in our security domain. In May, Microsoft published a system they called m dash, a a multimodal, agentic scanning harness.

21:21 More than a hundred specialized agents running across a mix of frontier and smaller distilled models, and they ran it against CyberGym, which is over fifteen hundred real world vulnerability task. They scored eighty eight point four percent, which is the top score on the public leader board. The next best score was eighty three point one percent. Now I want you to take note of who scored that eighty three percent because the entry that Microsoft beat was Mythos. The hundred orchestrated agents beat the frontier model,

21:51 not a different model, not a better model, just the same class of models deployed differently. So the harness, not the model, the ability to orchestrate, decompose a task, validation between steps, that is where our focus needs to be. Now I I wanna call it name something out loud because I I don't want you to feel like I'm having it both ways. I just cited a benchmark to make a point. But what I'm about to say is to tell you that benchmarks actually don't transfer to real world, scenarios. But both of these statements can be true. Here's why.

22:22 See, when it comes to why they don't, transfer to real world scenarios, m dash ran CyberGym in its level one configuration. What that means is that level one hands the test in system the entire vulnerable source code and a high level description of the vulnerability you're looking for. So read that result again with with that thought process in mind that Microsoft needed, more than a hundred orchestrated agents across an ensemble of front frontier and distilled models to win a benchmark that gives you source

22:52 code and tells you what you're looking for. Now picture that now now picture the real world target where you don't get anything. This is why benchmarks are valid, but not also valid at the same time because when you start talking real world, real world environments are messy. Real authentication has to be navigated, not bypassed. There's session timeouts. There's legacy fingerprints on a host system that nobody's inventoried inventoried for years. There's that back end that three acquisitions ago, somebody bolted on a system and the person who did that left in twenty nineteen.

23:25 See, there's no source code. There's no description of bug. There's no guarantee that there's even a bug at all. And it's that gap between a benchmark that hands you the answer key and the environment that hands you nothing. That's the single biggest reason that capability demos don't survive contact with production. So I'll dig into a little more of that here in a couple slides, but it's why the right mental model coming into this isn't just we bought and deployed a tool. It's thinking about the way you onboard an agent team the same

23:55 way you would onboard a human team. You know, in the world of offensive security, you scope it, you give it credentials, your rules, give it rules, you watch how it does on a target, you understand and trust before you trust it with one that you that's actually production and could have a bigger impact. And you build a feedback loop for when things go wrong. See, in a real world, you would never put a human team on a production team on a production target without rules of engagement. Same goes for agents, except now the enforcement has to happen at machine speed.

24:27 So after the model question, and the next question I always get asked when it comes to agentic AI solutions is guardrails. So he here's what guardrails actually look like in production. On the left is the shape of the answer. It's the policy layer that lives outside the model. Concretely, you know, a a runtime classifier that evaluates every action an agent is about to take against a specific customer's rules of engagement. It's across a swarm of agents rather than one at a time. And this is critical. It's the ability to stop an action mid execution,

24:59 not flag it, but stop it. Plus it has to have the ability to do the boring parts that that that really matter in practice. It's the scope boundaries, the rate limits, the target restrictions. They have to be held externally where a model doesn't have the ability to negotiate around them. And you need a full audit trail of what ran what ran and why. Because the the first question after an incident is always what happened. And the answer of we're not sure, that is not an acceptable answer. And I I have to be honest of where this came from because we

25:29 here at SYNNEC, we did not design all this upfront. It got built after our system left the lab and met real world customer environments. We we made a bunch of assumptions when we made this transition from the lab to production, and we found out which assumptions held true and which ones didn't. And this is how we develop a solution that produces results in real world scenarios. Now the right side why prompts aren't enough, it's because a prompt is not a control. And several reasons why prompts, they can be reasoned around.

25:59 An instruction inside a context window is an input to a model's reasoning. It's not a constraint. So a strong suggestion competing with everything else in that context. Also, agents' tasks decompose over time. Every task splits into a subtask. You get another opportunity for the original ten to get slightly lost. It's the old analogy of the telephone game. When you have one agent telling an agent telling another agent, things can change over time and drift. When you look at a swarm approach, one a swarm can amplify one bad inference.

26:33 This is a part that I think a lot of people underestimate is that parallelism is exactly what makes agent systems so useful and so powerful, but it's also what turns one bad judgment into fifty simultaneous bad actions. Lastly, you know, rules of engagements are contractual, not advisory. You signed something. The model didn't, and that asymmetry is a problem. Post hoc detection is too late. Detection is a fine strategy, but when you have minutes or even seconds at machine speed,

27:04 finding out afterward is too late. So here's the thing. You are gonna deploy agents. Maybe offensive ones, probably defensive ones. Almost certainly, you're already deploying operational ones that production touch production systems, and they may have be security related or have nothing to do with all with security. But every one of them has the same structural problem. It's a capable nondeterministic system taking actions faster than humans can review them. So whatever agents you deploy or whatever agents you purchase

27:35 and deploy, you know, offensive, defensive over operational, they need a policy layer that lives outside the model. And, you know, a control plane, what we're talking about answers one question. Can I stop it? There's another question we need to ask, and this goes back to provability being the new constraint is, should I believe it? And your peers have already answered and put a number to what believable looks like. So this slide here, this is peer date data. This is not a SYNNEXT specification. This is actually output from the report that we attached in

28:07 the chat that you can download. But this is data from two hundred security leaders that Omnia reached out to, and it it tells us what the market has already decided is table stakes. So let's start with the distribution on left hand side. When asked what accuracy an AI driven test and system would need to hit before they trusted alongside manual testing, the answer clusters hard in the eighty five to ninety four percent band. Sixty nine percent of respondents put the floor at eighty five percent above,

28:37 and almost nobody accepted below seventy five percent. Now hold that against the numbers that I gave you earlier with a nine point four percent false positive rate from Mythos and Palo Alto reporting roughly thirty percent from the frontier models across all their products, and then sixty two point four percent of reviewed findings were actually confirmed as genuine severe when we look at that mythos down funnel. And what this points out is there's a gap between reality and what the market demands. And that gap between the two is precisely where a lot of AI

29:09 security tooling is gonna fail when it comes to their first review. Now the other three figures on the right hand side, these are about trust rather than accuracy. And and they're the ones that if you sit on the the government side of the summit, you should pay attention to. Ninety three percent say guardrails are critically, are critical or important. That is near unanimous. And it's, it's actually interesting that people care more about bounded behavior than about being right. Fifty eight percent required transparent AI decision making,

29:40 meaning show me why. A finding that you can't explain is a finding you can't act on, and and, actually, it's a finding that you can't defend to a regulator. Fifty four percent require industry certifications and standard. Here's how I why I I present this slide and how you can use it. This is the bar to hold your own tool in, whether you build it or buy it. If you have a AI driven solution deployed in production right now and you can't tell me the false positive rate, then you honestly you don't know whether you're above or below this line your own peers just drew.

30:12 So we've got eighty five percent accuracy as a standard. We've got guardrails, transparency, certifications. This is a a demand in bar and and it's what table stakes for what your peer set. And the fair question is whether anybody actually, you know, delivers this in production. So let me show you our attempt here at and I wanna be specific about where humans are in this overall equation. So we're month twenty plus minutes into this presentation, and this is the first time I'm really gonna talk about what we

30:43 do at SYNNAC, and that is intentional and deliberate. My goal is not to come and just kind of tell you what we're doing at SYNNAC. I want everything to be able to stand on its own. So whether or not you speak to us after this presentation, I want you to have the knowledge that you can, again, though, do go and deploy against the agents you're looking at buying or building. So when it comes to SYNNAC, three numbers I wanna touch on first. First, fifteen hundred plus vetted security researchers on our SYNNAC red team bench. This is not an open sign up form. This is a vetted background check continuously

31:14 monitor monitored and assessed community. So second one, thirteen years in market, which means thirteen years of exposure to real world production environments, messy authentication, legacy hosts, the acquisitions nobody documented. We've been navigating these issues for well over a decade. Lastly, a a private controlled model, not not an open bug bounty, but known researchers define rules of engagement, full traffic attribution. Now let me connect those back to this presentation,

31:46 why I put this slide here rather than the front of the deck. Earlier, I mentioned that we you should onboard an AI agentic team the same way you onboard a human pen test team. This is what we've benchmarked our agentic solution against. Fifteen hundred researchers work in real targets for thirteen years. That gives us two things that you just can't buy quickly. First, a record of what real findings look like versus a false positive across thousands of live engagements. Second, people who can check the AI's work against it,

32:18 against reality when something goes wrong. It's something that a competitor who started twelve to eighteen months ago has a real world problem because they you just can't raise capital and license the same models and expect to have the same output when we've been doing this for thirteen years. So moving on to what we've delivered, we have a division of labor. See, we we believe that we take the symbiotic approach where it's the the AI with the human, not a fully autonomous alone, but humans alone itself won't address the problem either. So there's there's division of labor.

32:49 You know, on the AI side, you know, I mentioned our our solution, CERA, Synerg Autonomous Red Agent, handles the recon, the asset enumeration, and machine speed. It can handle that tier one stuff we talked about, the the known pattern detection. It can also do the the novel exploit, trying to find the past with privilege escalations, continuous coverage, you know, the autonomous triaging dupe dupe deduplication, which is where a lot of most of the false positive burden actually gets absorbed. But we still have humans in the process because as powerful as

33:21 AI is to AI is today, as capable as these frontier models are, AI alone is not adequate. The the the latest models, there's still certain limitations of things they can't do. See humans, they can handle horizontal chain and across disconnected systems. They they can understand business logic and the trust relationship attacks that AI is just not there yet. They humans have the ability to apply judgment on ambiguous findings. And I I just wanna be clear that the right hand column of

33:52 humans is a design choice. It's not a limitation we're apologizing for. It's because, as I mentioned, there's certain categories that humans are just generally better at today, and pretending otherwise would fail that accuracy standard we talked about on the previous slide. And this split isn't just, you know, our opinion. You know, you can read this in the Omnia survey results, but sixty four percent of leaders, prefer an agent led model with human oversight. And they're actually one point six times more likely to describe oversight as a permanent, not a temporary control once deployed,

34:24 which tells you that the people that are close to closest to this are less confident about a full autonomy, not more. So that's how we build it. Now I know you especially didn't come here to hear about architecture. You came here for something that you can use in your next conversation. So so here's how you can take this and apply it to your conversations, after this session. Five things. This can apply to whether you're building agents yourself or you're gonna buy them from a vendor. The first three on the left hand side,

34:54 these are controls that you put in place. The the second two on the right hand side, these are claims to verify before you sign anything. Number one, a validation gate. Nothing reaches a queue unreviewed. And I wanna be clear that while I just got finished that we believe in kind of the the AI and human combination, that doesn't always mean it has to be a human review, human validation before it gets to the queue. It's human in the loop where outcome matters. We don't want to lose the advantage of machine speed by

35:24 putting human stopgates at every, process, for the AgenTiKi. Number two, a runtime policy layer that can stop an agent mid action. That's the control plane that we described a few slides ago. It's whatever you deploy, ask where the enforcement lives. Does it live inside the prompt or outside the model? Number three, and this is specifically for, you know, the offensive agents that are deployed. Found and confirmed exploitable have to be two separate steps with two separate agents. A finding that says that looks vulnerable,

35:57 a finding says this looks vulnerable, confirmed exploitability means we made it work and we can show you proof. Here's the request that succeeded, the data that came back, the account that was taken over. If one system is doing both, that's like grading your own homework. And since severity is a driver in your mediation queue, if you're fixing in the order that the model felt confident about, that is not as ideal as fixing the order that's actually gonna reduce risk in your environment. So number four is we move to the right hand side, orchestration.

36:27 You know, not a single model. It's what's actually around the model. Is there a decomposition of the task? Are there retry processes? Is there cross checking? You know? And and in reality, you only also need to ask, you know, what is the token cost? Because the as we mentioned, Anthropics own published figure is that when you look at this orchestration approach versus single model approach, their figures were rough roughly fifteen times, increasing cost. And last, number five, benchmark literacy. Why benchmarks have value?

36:58 You also need to understand their limitations. So when you hear that benchmark score, ask the the context around that. Were they given source code? Were they given a description of bug? Were were there as a guarantee that bugs even exist at all? Like, is it that cyber level one problem cyber gym level one problem we discussed? It's understanding the difference between an impressive number and a relevant one. So ask your engineers or any vendor you're talking to these same five questions. These answers are gonna tell you where the harness ends and

37:28 where the hype begins. And I wanna be straightforward with you. I think we at SYNNAC answer these questions well, but I've deliberately not framed these questions as a reason to pick us because I wanted a a version of this that's useful to you, a framework that you can apply across the board because that's the one you can take away. If you ask us these five questions, you don't like that answer, that's a legitimate result, you should act on it, which brings me back to where we started. Let's go back to two thousand four Metasploit.

37:59 The organizations that came out of the transition ahead were not the ones that, you know, increase their perimeter defenses. They weren't the ones that could react the fastest to every new exploit that was announced. They were the ones that looked at their program and decided to evolve and reshape what they were doing. As they stopped treat treat and security test as event that happens once or twice a year, and they started looking at how can we do this on a continuous basis. So look at today, Mythos, GPT five point six cyber, the m dash.

38:30 You know, it's we're at the same inflection point. The axis curve has collapsed again, and the question is not whether to respond, it's whether you respond before your adversaries. And one practical note about the environment that we're making these decisions in, a recent report from Forrester projects that enterprises will defer about a quarter of their planned AI spend to twenty twenty seven. As CFOs, they start asking the ROI evidence that most teams aren't able to produce yet. And note that word deferred is not necessarily cancel. It's just push off.

39:00 What this means is that the AI security investment that you're looking to propose in the next two quarters are gonna face a much harder business case and validation than the same proposal would have faced a year ago. So make sure you bring the numbers as you propose solution. It's it's why I dug in so much the evidence today rather than the hype so you are, have the ammunition. If you wanna prove this works in a regulated environment, as I mentioned, we attach the Iberia cards case study in the chat there in financial services,

39:30 their European regulatory regime. It's very useful read on what governments and how AgenTek AI operates in such a practice. So let me close with one question I I'd like you to be able to answer. How much of what you found last year could you actually prove? Not not findings you generated, not the outputs of the scanners, but how much could you actually stand behind well enough that you're willing willing to spend a change window on them? That number, not not the number you found, is what your program actually delivered.

40:01 Everything else was volume that you just paid to produce. So let's just realize, find and stop being the hard part. Proven is where we need to focus our efforts on now. If you would like to learn more about that, we would love for you to reach out to us so we continue that conversation. But again, thanks for taking the time to sit through the session and enjoying the rest of the webinar.

No lines match that search.