Meet the experts who power Synack’s strategic security testing platform. Our Synack Red Team unites over 1,500 of the world’s most skilled and trusted security researchers, who work with patented technology to deliver best-in-class offensive security testing on a continuous basis.
The AI Episode: Experts Share Security Insights on LLMs and GenAI
Published:
Updated:
Duration:
15 min
Episode summary
Hear from this season’s guests for their thoughts and predictions on AI and cybersecurity. We took sound bites from a range of WE’RE IN! interviewees, whose opinions on AI vary from thinking it’s overblown to being cautiously curious. Tune into this episode to better understand AI’s seismic effects on the infosec industry.
Listen to the episode
S3 E18 · The AI Episode: Experts Share Security Insights on LLMs and GenAIWE'RE IN! Podcast · 15 min
Key takeaways
Fundamentals before AI-specific security Ryan Kazanciyan warns that many organizations still struggle with authentication, authorization, and cloud security, and that jumping straight to securing AI leaves those gaps open.
Attackers will use AI agents Mark Kuhr expects attackers to use AI to pick targets and entry points, and multi-agent frameworks that use tools to run recon and throw exploits. He says defenders should be using AI now.
AI can map data and perimeter Kevin Tambascio sees machine learning helping find excessive permissions, retention policy issues, and unknown applications exposed on an organization's perimeter.
Offensive security still needs humans Tennisha Martin points to research showing AI stops being effective at a certain point for critical thinking and problem solving, which is why penetration testing keeps a human in the loop.
AI platforms have ordinary vulnerabilities Anand Prakash notes that AI platforms leak large numbers of API keys across GitHub, GitLab, and Bitbucket every day, showing that traditional issues remain.
Topics covered
AI & LLMs
Notable quotes
A lot of organizations are still struggling with really basic fundamentals around authentication and authorization, around cloud security.
You should be using AI for defense, especially because the offensive side of the house is gonna start cranking up the use of AI for offensive operations.
…the AI platforms which are built right now themselves have traditional vulnerabilities.
I am a self declared skeptic when it comes to AI.
Questions this episode answers
How should organizations prioritize AI security?
Ryan Kazanciyan says to keep AI risk in proportion to other initiatives and focus first on fundamentals like access to data, authentication, authorization, cloud security, and visibility. Read this section
How will attackers use AI in offensive operations?
Mark Kuhr expects AI to help attackers select the best targets, find optimal entry points and exploits, and use AI agents that run tools and iterate on techniques together. Read this section
What are adversarial machine learning attacks?
Amy Chang mentions query based attacks and data poisoning attacks, with real world impacts such as data leaks, inaccurate medical diagnoses, or manipulation of financial markets. She recommends secure by design principles and frameworks like the AI risk management framework. Read this section
Will AI replace penetration testers?
Tennisha Martin does not think so, because research shows AI stops being effective at critical thinking and problem solving and a human in the loop is still needed. Read this section
Will AI replace security analysts?
Anthony Newman says the key is that AI is not replacing analyst jobs but making them simpler, leaving analysts the hard work of thinking and reasoning. Read this section
Hello, and welcome to WE'RE IN, a podcast that gets inside the brightest minds in cybersecurity. I'm your host, Blake Thompson-Heuer. And today, we have a bit of a special edition in store for you. There's no denying that AI is having a moment. Companies in seemingly every industry from Silicon Valley juggernauts and small town car dealerships to financial service providers and independent app developers are all experimenting with generative artificial intelligence offerings such as ChatGPT. The same thing is happening in the security industry. Gen AI has been a topic of conversation across our third season of this podcast. So in this episode, we've collected relevant comments from many of our guests. This is how leading infosec professionals view the promise or at least the potential of AI. But before we get into it, here's a quick word from our sponsor, Synack.
Confusion in the marketplace about which is best method of security testing is so real. Bug bounty, pentesting, pentesting as a service, what do they all mean? We break it down in our latest playbook, navigating the security testing landscape to demonstrate the strategic value of third party security testing. We cherry pick the best elements across the security testing market to incorporate into a strategic, comprehensive pentesting solution, the Synack platform. Learn more at synack.com/playbook. That's synack.com/playbook.
We'll start with Ryan Kazanciyan, the chief information officer and chief information security officer at Wiz with a conversation about securing the AI future.
Across my entire career, I'm not sure I can think of a single technology that has had the degree of disruption or at least at this pace as AI and AI enabled services. The first thing is I always like to focus on fundamentals. And so prioritization is important. It is as an organization, even if you're not building software, if you're using services that are bringing AI features in, to understand what is your risk of exposure versus all the other like security and attack surface management initiatives you have. A lot of organizations are still struggling with really basic fundamentals around authentication and authorization, around cloud security. And if you don't get those fundamentals right, and you just jump straight to the, like, problem of the day, which is securing AI, you're gonna leave a lot of things open. So keeping AI risk in proportion and relative to all of your other initiatives is important because if you're not managing access to data, if you're not managing the security of your systems or infrastructure, if you don't have like good visibility and coverage, AI is only one facet of a number of risks that you're probably exposed to and need to manage.
Ryan wasn't the only guest concerned about securely incorporating AI into an organization's tech stack. Microsoft's Chief Security Advisor for Europe, The Middle East, Africa, Sarah Armstrong-Smith, raised similar concerns. Here she is.
One camp is looking at the art of the possible, how do I embrace this new technology, lots of different case studies. We've seen huge number of businesses who are standing at R and D and innovation centers, all very cool, thinking about what does the future look like. That being said, when I talked about some of the conversations that I'm having with CISOs and other technology leaders, there's a little bit of a nervousness really about do we really understand this technology? And even if we kind of think about in the last year, we've talked about what is biases, you know, is hallucinating, is mixed stuff up. We don't really understand the consequences of how this technology is being utilized.
Next, we'll hear from doctor Mark Kuhr, the CTO and cofounder of Synack, who said he believes AI tools will be the latest area where defenders and attackers continually try to one up each other. Mark isn't alone in thinking AI powered or at least AI supplemented products will become increasingly popular over the next few years.
It's a really interesting cat and mouse game that's going on right now with the developers of the applications and the researchers. What we're finding is, there's a lot of edge cases that are not being handled yet as people are very new at developing applications around these AI ecosystems. And, if you're not using this for defense, you should be. You should be using AI for defense, especially because the offensive side of the house is gonna start cranking up the use of AI for offensive operations. And that could be selecting the best target, going through a lot of data on an organization's assets and finding the best target and the optimal entry points, the optimal application of exploits, the optimal way to come at an organization and exploit it quickly. And then the AI agents, you know, can also get to the point of using tools.
So they're also using tools to throw exploits, to do recon, and gain access. And then they're... If you have a multi agent framework, you can even start to get these agents talking to each other and iterating on their techniques for gaining access. It's certainly an exciting research angle. If you read the latest research on this, there have been papers published on AI agents finding vulnerabilities and breaking into applications. This is just the start, but I have a feeling, you know, we're gonna see this explode in the next couple years.
Cleveland Clinic Director of Cybersecurity Data and Application Protection, Kevin Tambascio, also said that artificial intelligence is paving the way for more products and services to see it incorporated.
I mean, I don't know how many times a day I get a message from somebody that, yeah, they're putting AI in their product. Sometimes it doesn't make sense, but I think when it comes in my area for data protection, as well as application protection, there's definitely some great possibilities. I mean, when you think of data protection, machine learning is great at consuming data. Well, guess what? We have tons of data, right? And so you can start to apply those principles to understand the landscape, understand where are there excessive permissions? Where are there policy issues with retention? We can start to look at how information is being shared. Should it be shared externally? Right? And I think these platforms can help us. And then same with app security. I see a lot of potential for enhancing our ability to understand our perimeter. I think like many large organizations, it's challenging to understand what are all the applications, what are all the things exposed on your perimeter?
What's your attack surface look like externally? And then I think we also see AI tools that can help with pressure testing those applications, looking for gaps and maybe augment security researchers who can find maybe the more complex things, also helping developers write more secure code.
Disruptive technologies like Gen AI often follow a familiar pattern. Small companies experiment with promising advancements, which are then adopted by larger companies, which then prompts regulatory bodies around the world to seriously consider the ramifications of these newly popular offerings. The same's happening with AI as we discussed with Amy Chang, then a senior fellow of cybersecurity emerging threats at the Institute, who is now an AI security researcher at Robust Intelligence. Amy and I also touched on some of the broader implications of widespread adoption of AI tools.
Core focus of our team is examining cybersecurity issues as they pertain to how to think about them, how policymakers could legislate or think about regulating cybersecurity, as well as emerging threats such as, well not threats necessarily, emerging technologies that have the potential to manifest into threats, artificial intelligence, quantum computing, and things like that. One area that we're focusing on very deeply lately is the nexus between artificial intelligence and cybersecurity. I think we hear a lot about how AI could potentially lead to doom and gloom type scenarios, but what we wanna do in our program is also highlight how AI can benefit cybersecurity and how we can embrace its many applications in security.
We went into things like query based attacks, data poisoning attacks. I would say reading it, you can have a diagram of how those attacks would occur. We also discussed the real world impact of adversarial ML attacks, which could include data breaches or data leaks, inaccurate medical diagnoses, or potential even manipulation of financial markets. As you said, Blake, very scary stuff, But we also go in and talk about how you can protect against those those types of attacks too, such as incorporating this concept called secure by design principles, which is just basically bake security into the design of a specific product, all the way until like, you know, utilizing security frameworks such as the AI risk management framework, or even, you know, ways that we could support new entrants that can challenge the AI giants and make sure that, you know, all standards are are kept high and and that any sort of risks are are kind of taken into consideration.
The cybersecurity industry's budding interest in Gen AI could also affect workers. Anthony Newman, executive director of the research and education networks information sharing and analysis center, told me he thinks this could be for the better.
AI could actually detect anomalies that that are, you know, somewhat actionable pretty quickly. So you could... I I I don't think any technology is really there yet commercially. They're getting there. But I think that's what is going to be necessary to to move forward in a way where we can trust not only AI, just technology in general. But I think that the key for me is is it's not replacing those analyst jobs. It's it's making their jobs a lot simpler, and allowing them to do the the hard work of of thinking and and reasoning versus taking some of those large language model... Models and some of the data that it can ingest. To me, that's kind of the dumb side of AI, but it's been... It does it really, really well. Yeah. We'll we'll see. I think we're still probably a year away, but, know, then again, I thought the video was going to be a couple years away, and they just put out a release, I think, a week or two ago that was pretty good.
Black Girls Hack founder and executive director, Tennisha Martin, was similarly optimistic about AI's impact on cyber professionals, especially offensive security specialists who really need to think outside the box to help organizations learn how to defend themselves from real world attackers.
People say, you know, when you're talking about AI, they'll say, you know, hey, AI is gonna one point take all of our jobs. I think that penetration testing, ethical hacking is one of those things where we don't have to worry about that taking because a lot of the research that they've done thus far shows that there has to be some sort of like, they call it human in the loop where it's basically... There's a certain demarcation point where... At which AI stops being effective when it comes to doing things like critical thinking, problem solving. At what point do you have a human take a look at this?
But that isn't to say defenders will be easy to replace either. PingSafe founder and CEO, Anand Prakash, who has since joined SentinelOne to work on cloud native security, told me that we're still early in the Gen AI life cycle. These artificial intelligence products aren't ready to take anyone's job, and many cases, they still need someone to figure out how to securely operate them in the first place.
I have seen couple of good companies coming out in AI who are trying to solve for AppSec, Red Teaming. For example there was an AppSec company which is trying to analyze behavior across the entire lifecycle and trying to surface issues very early in the pipeline. But I think it's still early. Has to be innovation which has to happen on this front for now. But right now the state if you talk about AI, the AI platforms which are built right now themselves have traditional vulnerabilities. So I feel like there is a scope for improvement. If you look at the number of API keys being leaked by these AI platforms all across GitHub, GitLab, Bitbucket every day, the number is large, huge.
Finally, it's important to remember that Gen AI didn't just spring into existence. The field draws from computer science and mathematics research dating back centuries, as Mitel CISO Bill Dunnion reminded me.
First off, I I am a self declared skeptic when it comes to AI. I'll put that disclaimer out there right off the bat. I've done a lot of research on AI and the origins, I guess, probably, and I know I have to get over it, but to me it's the misuse of the term artificial intelligence. For the geeks on the line here, the fellow geeks on the line, because yes, I'm one, and the origins actually go back to the Second World War and Alan Turing. All the algorithms we're using today have their foundation and origins going back to Alan's work back in the 30s and 40s. And there's clear levels, right? And artificial intelligence is the top or the fourth level. Everything else is machine learning. And so that's really the the realm that we're in.
who shared their thoughts with me, and stay tuned as we round out season three with some more stellar experts. Until next time, thanks for listening. If you liked what you heard today, I hope you'll give us a five star rating and review. It's a big help. And please share this episode if you know anyone who could appreciate a little info sec wisdom on their morning commute. We have a whole catalog of episodes well worth a listen, so you may wanna check out past interviews as well. Finally, if you know someone who might be a good fit to appear on the podcast or have any comments or feedback, drop us a line at we'[email protected]. That's s y n a c k dot com. Until next time.
WE'RE IN is brought to you by Synack. If you're looking for on demand continuous access to the world's most skilled and trusted security researchers, you can learn more at synack.com. Synack recently launched its Empower Partner Program so that partner organizations can more easily offer the Synack Pen Testing platform to their own customers. This approach helps optimize Synack partners' technical competencies and allows them to better integrate Synack into their portfolios. It's a way that partners can win new business by adding continuous best in class solutions to cybersecurity, cloud, and DevSecOps offerings. Synack partners with organizations around the world to make them safer, more resistant to cyber attacks, and more capable of finding and fixing dangerous vulnerabilities before attackers are able to exploit them. Learn more at synack.com. That's synack.com.
