"Across six recently disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models," Keith Hoodlet, director of security research at 1Password, wrote in a blog post.
1Password's Off-by-1 Labs experiment
Researchers at 1Password's Off-by-1 Labs set out to evaluate whether large language models (LLMs) can autonomously produce security patches that reliably fix real vulnerabilities. The team used two "frontier" models — ChatGPT 5.5 at "medium" effort and Claude Opus 4.8 at "high" effort — and generated a total of 6,080 patches across six recently disclosed CVEs. The paper documenting the work lists Axel Mierczuk, Spencer Michaels, and Keith Hoodlet as authors and presents both quantitative outcomes and a released evaluation harness.
Patch outcomes and the FLAWED finding
The experiment's headline result is stark: the average success rate for generating a patch that fully resolved the vulnerability without materially changing application behavior was just 26.0 percent. The researchers devised the acronym FLAWED — Fix-Like Artifacts With Embedded Defects — to describe the category of automated patches that appear to fix a problem at first glance but carry embedded defects or incomplete remediations.
Broken down, the AI-generated patches fell into these outcome buckets:
- 26.0% — fully resolved the vulnerability without materially changing application behavior;
- 20.1% — fixed the original issue but altered application behavior (for example, changing "allow list" logic to "deny list" logic);
- 2.3% — fixed the issue while introducing new security issues;
- 49.3% — failed to fix at least one existing exploit path;
- 2.2% — failed to fix the vulnerability and introduced a new exploit path.
Among the patches classified as either successful-and-clean or successful-but-changing-behavior, the authors rated more than a third as "fragile": repairs that guarded against a particular exploit (for example, escaping certain input characters) but did not address the underlying root cause.
Audit-ready is a season. It shouldn't be.
Evidence in spreadsheets, controls drifting between audits, frameworks multiplying on flat headcount. Nubivance runs continuous compliance on Rapid7 Cyber GRC - SOC 2, HIPAA, ISO 27001, PCI, CMMC.
End the scrambleGuidance quality changed outcomes dramatically
The research shows that the quality of initial guidance materially affected LLM performance. When given correct guidance, LLM-generated fixes reached a 65.0 percent fix-success rate. With no guidance, the fix-success rate fell to 50.4 percent. And when given incorrect guidance, LLMs performed worst — about a 15.2 percent fix-success rate. The authors contrast this with human developers, arguing that humans "have a good chance of catching misleading information as they reason through vulnerable code."
Cost per patch, and the hidden cost of review
Viewed in isolation, some LLM-generated patches look inexpensive: the average successful, clean patch cost just $6.74, a figure that includes the cost of failed attempts. But the researchers caution that cost-benefit calculations must include the supervision burden required to make those patches safe and useful. As the paper warns, "the cognitive load imposed by reviewing a mountain of mostly-incorrect, similar-yet-subtly-different LLM-generated vulnerability patches will likely result in engineers spending more effort than would be necessary to understand and patch vulnerabilities themselves using standard LLM-assisted coding techniques that keep the human operator in the driver’s seat."
The paper's conclusion is categorical: "[T]he expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin." In response, the authors released a patch evaluation harness under the name FLAWED for organizations to evaluate the effectiveness of candidate fixes.
What this means for technologists, procurement leaders, and end users
- Technologists and security teams: The results suggest LLMs can assist with patch ideas but cannot be trusted to operate autonomously. Teams will need to continue code review and deeper analysis to detect fragile fixes and pathways left unaddressed.
- Enterprises and procurement leaders: The low per-patch dollar figure masks a likely supervisory cost. The paper implies organizations should budget for expert review and validation rather than assuming autonomous LLM patching reduces headcount or external spending.
- End users and the general public: Apparent fixes rolled out without human review risk behavioral changes or new vulnerabilities. The presence of fragile repairs means an application may appear secure for a specific exploit while remaining vulnerable in other ways.
The Off-by-1 Labs experiment does not close the door on LLM assistance; it reframes the calculus. Correct guidance can substantially raise success rates, and automated patching can produce low-cost candidates. But the paper's data — 6,080 patches, a 26.0 percent clean success rate, and many fragile or incomplete fixes — underlines a simple, actionable caution: for now, human attention remains essential before AI-generated patches reach production.
Source: The Register — AI struggles to patch vulns without adult supervision
