"The strict methodology was never the good bit," writes Daniel Thatcher, a security research engineer at Intruder, drawing on twelve years of penetration‑testing experience and recent work building Intruder’s AI pentester.
Structured systems versus free agents at Intruder
Thatcher describes a practical experiment: early iterations of Intruder’s AI agents were built around rigid, role‑based methodologies — each agent assigned to hunt a specific vulnerability class. That structure produced predictable failures: agents missed issues outside their remit, or force‑fit findings into the category they were told to look for, creating confusing reports. Conversely, completely unconstrained agents also failed, because they could not break down or manage large applications reliably.
The compromise Intruder settled on is neither checklist nor chaos. Agents are given broadly defined roles and "stricter guardrails to stop them causing damage." That design tradeoff aims to let agents follow unexpected traces through an application while preventing destructive or out‑of‑scope activity.
White‑box source access: what agents can and cannot do
Intruder’s AI pentests operate in white‑box mode: agents receive the source code and "work through the entire codebase," reconstructing pages, application flows, and request patterns to a level of detail Thatcher says a human cannot reach in a typical engagement. The agents build an understanding of application purpose and access control without being told which users should or should not reach admin functionality.
But there are limits. Unlike a human tester, an agent "can't ask you for anything you didn't already give or tell them." Humans can query contacts to clarify ambiguous features, confirm scope, or request permission to run intrusive checks; Intruder intentionally made agents cautious to avoid stepping beyond scope.
This site is the portfolio.
OSINTSights runs on Cloudflare Workers, D1, R2, and Vectorize, with an AI pipeline on Hetzner ARM. Nubivance designed, built, and operates it. We do the same for clients.
See what we buildCoverage and variability: neither approach finds everything
Thatcher is blunt: "Neither finds everything." The freedom that enables an agent to discover a complex authentication bypass — an issue that had survived five years of human pentests — also makes AI results less predictable. Agents decide where to dig, and their paths will differ from run to run.
That variability mirrors human testing. Human findings vary by tester experience, specialism, and incidental knowledge; checklists are not a panacea. Intruder acknowledges consistent coverage remains a work in progress, and notes that human pentest reports have always carried a disclaimer acknowledging fallibility.
Reporting, risk scoring, and remediation at line level
Thatcher highlights where agents already outperform humans: remediation guidance. With access to source, agents can produce precise fixes — "the line of code to change, and what to change it to" — and include mitigating controls or temporary workarounds. That depth appears to accelerate fixes in practice: a referenced customer with 78 issues remediated them "within a few days," an outcome Thatcher says he had never seen as a human pentester.
Agents also generate executive summaries that Thatcher finds genuinely useful. Where they fall short is risk scoring: agents tend to rate issues more highly than Thatcher would, "partly because they can't have the conversations with you that would bring a rating down, and partly because LLMs err towards caution when you start talking about cybersecurity." Intruder reports it is working on calibrating that behaviour.
What this means for technologists, procurement leaders, and adversaries
- Technologists and security teams: AI agents’ white‑box, code‑level analysis can surface non‑obvious vulnerabilities (the example of a long‑standing auth bypass) and deliver remediation steps that accelerate fixes. Teams should expect more frequent reports if they adopt automated runs "whenever you want."
- Procurement leaders: Thatcher notes a cost argument — "for what a single human engagement costs you can run four AI pentests" — and operational advantages: "no lead time, no scoping calls, a fraction of the cost." Buyers weighing cadence against depth will need to decide whether regular AI runs complement or replace periodic human engagements.
- Adversaries and threat actors: Thatcher warns implicitly that uncommon, hard‑to‑predict vulnerabilities matter because "just because they’re not common, it doesn’t mean that an attacker can’t find them too, especially now AI is being used to enable attacks more." The finding underlines the urgency of closing windows of exposure between human pentests.
Thatcher’s conclusion is pragmatic: AI pentesting and human pentesting are different tools with different strengths. Intruder’s agents can read full codebases, follow unexpected trails, and produce executable remediation at scale and speed — and they can be run far more often than a traditional engagement. But agents cannot ask clarifying questions, they can be variable in coverage, and they currently tend toward conservative risk scoring. Intruder says it is iterating on coverage and scoring; the tradeoffs Thatcher documents suggest organizations must decide how to mix human judgment and automated depth to manage risk.
