AI: Both a Help and a Hindrance for the Public Sector

Last week, we hosted the Synack Security Symposium in Washington, D.C. It was an open forum for Synack customers, partners, and the local cybersecurity community. Wade Lance, Synack’s Global Field CISO, led a lively discussion on cybersecurity in the age of AI.

Several themes came up: the advantages of AI, interoperability, and the value of human-led testing. They’re worth attention from government CIOs and CISOs. The industry view of AI has shifted — from a novelty to test, to powerful tech with both major risks and mission rewards.

AI Poses Multiple Attacker & Defender Advantages

AI helps find vulnerabilities faster, for blue teams and adversaries alike. By 2025, AI may fuel “faster-paced, more effective and larger scale” cyberattacks, according to a UK government study reported by the BBC. Cyber criminals can research targets far faster with AI, sharpening the speed and accuracy needed to exploit vulnerabilities and misconfigurations.

At the same time, defenders need training on when to use public or private AI engines — and when not to. The risk of data leakage is real. Leakage can present inaccurate or outdated information as fact. It can also expose sensitive data like PII, which can lead to identity theft.

Defenders and their managers will also need sanitization scripts and other safeguards. These keep public AI engines from exposing PII, CUI, or FOUO material that platform users or training models could later reuse. Train involved team members consistently, with extra care for sensitive, mission-specific datasets that may not be right for AI.

Finally, AI deployments like chatbots and enhanced search engines add their own risks to the attack surface — including common flaws in the OWASP LLM Top 10. Organizations should account for these tools when planning their security testing, or risk handing attackers new ways in.

Sanitization of Scripts is Critical

Sanitize generative AI prompt data for accuracy and clarity. Make sure text-to-speech programs aren’t vulnerable to unwanted outputs.

You can curb prompt vulnerabilities from the start with presence penalties, max tokens, and the right sampling. Testing for prompt injection also reduces user incidents.

Interoperability is a key consideration for purple teams leveraging AI for speed

Teamwork between red and blue teams is breaking down long-standing silos across cyber and IT. Offensive (red) teams have historically tested, identified, and triaged vulnerabilities. Defensive (blue) teams patch them, and also hunt and investigate incidents. AI-powered technology — from scanning tools to SIEM and SOAR — now makes it common for modern purple teams to correlate vulnerability reports with logging data and threat intel.

For this collaboration to continue, data source analysis must stay consistent — a big ask for government agencies. Modernization can quickly connect legacy IT and its workflows and policies. AI will be key to the speed needed to model and visualize the outputs. And continuously testing how these systems are configured is critical to keeping vulnerabilities low.

AI can aid the Secure by Design, Secure by Default mission

This year, Jen Easterly and CISA expanded the call to “shift left,” urging tech companies to ship products that are Secure by Design, Secure by Default. As DevSecOps teams grow across critical industries, product and engineering teams are pushed to find vulnerabilities and patch code earlier — especially in small-batch development. That frees security leaders to invest in testing legacy IT and the connection points between older systems and modern cloud- and SaaS-based systems.

Humans remain the most important part of offensive and defensive solutions 

Keeping government cyber talent engaged and motivated is critical to long-term success. So is empowering every agency employee to stay vigilant. As MFA blunts attackers’ once-easy wins, social engineering will keep growing. One major tactic is vishing, or voice phishing, where attackers use a convincing AI-powered phone call to gain access to a system.

Vishing has become a real concern lately — just ask MGM. Business email compromise isn’t new either, but AI-enabled, fast decision-making makes it easier to attempt. Encourage staff to flag suspicious emails and to never share details by email with unknown parties.

Taken together, the range of talent exploring AI for government use is impressive, and the gains for productivity should be significant — inside and outside the software development cycle. But the potential impacts keep changing, and they’re enormous. For more than 10 years, Synack has stood with security teams, augmenting red team efforts for hundreds of organizations. The result: fewer vulnerabilities and a more secure attack surface.

Synack is gearing up to test AI deployments on customer attack surfaces. To learn more, visit our Pentesting AI and Large Language Models (LLM) page. For a deeper look at Synack’s pentesting and reporting for LLMs, contact your Synack success representative or book a demo.