AI Pentesting: What Buyers Should Ask Vendors to Prove

Ask an AI pentesting vendor to prove what was tested, what its findings demonstrate, where human experts contribute, how testing is controlled and what supports its performance claims. The answers should help you decide whether the testing will produce evidence your security team can use.

As a CMO, I believe we have a responsibility to make those answers easier to find. Buyers should be able to challenge our claims and understand their limits before making a commitment.

AI gives attackers greater speed and scale across an expanding attack surface. Security teams need testing that keeps pace, with the expertise to investigate complex attack paths and judge their business impact.

Words like “autonomous,” “agentic” and “continuous” need specific explanations. A buyer should leave the conversation knowing what happens during a test and what the resulting evidence supports.

Tim Nordvedt’s questions to ask a penetration testing vendor cover the broader engagement and contractual commitments. My focus here is the standard of proof behind AI pentesting claims, and the questions I would encourage buyers to ask every provider, including us.

1. What was actually tested?

Start with the scope behind the claim. Which applications, APIs and infrastructure were tested, with what access? Ask which environments and authenticated workflows were included, from pre-production to live production where authorized, and which were excluded.

An asset appearing in an inventory tells you something different from evidence that its workflows were examined. Ask the vendor to distinguish assets discovered from assets tested, and to show the depth and limitations of that testing.

The same discipline applies to “continuous.” Ask what runs continuously, how changes trigger testing and how often findings are reviewed and validated.

I would expect the explanation to make untested areas visible. That gives the buyer a clearer basis for deciding what additional coverage is needed.

2. What does a finding actually prove?

Ask to see a sample finding and its supporting evidence. Does it identify a potential weakness, demonstrate exploitation or show how the weakness contributes to a wider attack path? The report should make the level of proof clear.

Consider an access-control issue in a customer application. A useful finding would explain the permissions used, the action attempted, what access was obtained and which data or business process was affected. It should also state what the testers did not attempt.

Our Learning Center article on exploitability validation explains the technical distinction. Phil Whitby also examines the engineering required to turn AI-generated findings into proven risk.

For the buyer, I would bring the discussion back to the decision: can the application owner understand what needs fixing and why? Ask how remediation is retested, so the evidence can establish whether the fix worked.

3. Where do human experts contribute?

“Human in the loop” deserves a specific explanation. Ask who directs the testing, when experts intervene, how they investigate ambiguous results and who takes responsibility for the final finding.

I would also ask what experts do beyond reviewing output. Can they pursue business-logic abuse, investigate chained attack paths and challenge the assumptions that shaped the automated testing?

At Synack, our AI + human penetration testing approach brings Sara AI Pentesting together with the vetted researchers of the Synack Red Team. Agentic AI extends speed and reach, while experts provide direction, creativity, business context and validation.

Our promise is to show customers what is exploitable, what matters, and what to do next. Buyers should ask us how the testing and evidence deliver on that promise in their environment.

4. How is testing controlled?

Before testing begins, there should be agreement on authorized activities and boundaries. NIST’s definition of rules of engagement places those guidelines and constraints before the start of a security test.

For agentic AI pentesting, ask how the boundaries are enforced during execution. What prevents interaction with an out-of-scope asset, which actions are prohibited and how can testing be stopped?

Ask what activity is logged, who can review it and how customer data is handled. Our guidance on evaluating AI pentesting guardrails provides a starting point for those discussions.

Production testing deserves an explicit explanation of operational constraints and escalation procedures. I would ask the provider to demonstrate its controls and explain the limits of the assurances it offers.

5. What evidence supports the performance claims?

A demo is a useful introduction. Ask for the methodology behind any claim about effectiveness, including the environments tested, the definition of success and the time, access and human assistance provided.

If a vendor presents a benchmark, ask whether the comparison used equivalent conditions and whether the results can be independently examined. Include missed findings and false positives in that discussion, alongside successful discoveries.

Customer research can explain priorities and adoption. Technical evaluations can examine performance under defined conditions. We should be clear about which evidence supports each claim, so buyers can judge its relevance to their own environment.

The standard we should set

I believe marketing earns trust when it makes claims easier to evaluate. That means publishing supporting evidence, explaining limitations and giving buyers questions they can use with any vendor.

The same standard should apply across full-spectrum offensive security, whether the customer needs a deep assessment, recurring testing or an on-demand engagement. The outcome is confidence about what is exploitable, which risks need attention and what action to take.

We should welcome that scrutiny of our own messaging. Before the conversation moves to the next feature, buyers deserve a clear answer to one request: show me what your testing proves.