"This isn’t unique to GitHub," wrote Jarom Brown, senior product security engineer at GitHub.
GitHub tightens the definition of a “complete” bug report
GitHub says it is changing how it defines a “complete” bug report after a marked rise in AI-assisted submissions over the past year. The company told its users that many of these reports arrive without proof of concept, lean on unrealistic attack scenarios, or target issues already listed as ineligible — making it hard to separate signal from noise. Brown warned that some programs are "grappling with the same challenge, and some have shut down entirely."
GitHub did not propose banning AI-generated reports outright. Brown called AI a "force multiplier" when used correctly, but he emphasized the longstanding standard the company expects: "An AI-assisted finding that’s been verified, reproduced, and submitted with a working proof of concept is a great submission. An unvalidated output submitted as-is without reproduction or demonstrated impact is not."
Cloudflare’s experiment with Anthropic’s Mythos
Grant Bourzikas, chief security officer at Cloudflare, said triage and proof-of-exploit work have always been among the hardest parts of vulnerability research — and that AI vulnerability scanners have “made it worse.” He pointed especially to the behavior of AI tools that are designed to give users what they ask for and therefore produce speculative findings that demand human follow-up.
Cloudflare tested Anthropic’s Mythos on 50 of its own code repositories. Bourzikas described Mythos as “a different kind of tool doing a different kind of work” and credited it with meaningful reductions in false positives. He highlighted two Mythos capabilities that stood out: the ability to chain exploits together and the ability to generate its own proof-of-concept code to confirm exploitability. Those features, he said, helped Mythos demonstrate impact where older models could spot a bug but not show how it could be exploited in real-world conditions.
At the same time, Bourzikas cautioned that AI tools scanning software written in memory-unsafe languages (the example cited was C and C++) are far more likely to generate false positives versus scans of code in memory-safe languages like Rust. He wrote that speculative outputs are a reasonable bias for an exploratory tool but “a ruinous one for a triage queue, where every speculative finding spends human attention and tokens to dismiss, and that cost compounds across thousands of findings.”
Daniel Stenberg and the curl stress test of Mythos
Daniel Stenberg, lead developer for curl, described his experience with Mythos Preview after receiving a scan of 178,000 lines of curl code. Mythos flagged five "confirmed" vulnerabilities; human follow-up found four of those to be false positives or to have no security impact. The single remaining finding was a low-severity flaw that Stenberg said will be fixed in a regular June update.
Stenberg praised the broader impact of AI on security but concluded that Mythos so far appears only “a bit better” than previously released models. He wrote that “the big hype around this model so far was primarily marketing,” and that he saw “no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos.” He also noted that the earlier flood of low-quality AI-generated reports has tapered off significantly since March as models have improved.
Separately, the reporting noted that Anthropic reportedly chose not to release Mythos to the general public because it was so powerful at finding vulnerabilities.
What this means for bug bounty programs, security teams, and open-source maintainers
- Bug bounty programs: Programs must raise their submission standards or face a growing triage burden; GitHub’s move to tighten the “complete” report definition is an explicit attempt to reduce the noise that can force programs to curtail or even end participation, as Brown warned.
- Security teams and CSOs: Teams like Cloudflare’s are experimenting with frontier models such as Mythos to reduce false positives by requiring exploit chaining and proof-of-concept generation, but they still face more speculative outputs from general AI tools — especially when scanning memory-unsafe code.
- Open-source maintainers: Developers such as Daniel Stenberg are using AI scans as another source of potential issues, but their hands-on reviews remain the definitive filter; Stenberg’s curl scan turned five flagged items into a single low-severity fix scheduled for June.
Where the facts leave us
Organizations now face a paradox: newer AI tools can produce higher-fidelity exploit demonstrations, yet the overall volume of AI-assisted reports has multiplied the human effort required to triage speculative or irrelevant findings. GitHub’s call for validation — verified, reproducible findings with working proofs of concept — is the practical response named in the reporting. Cloudflare’s tests suggest certain frontier capabilities (exploit chaining and automatic proof-of-concept code) can blunt false positives, while curl’s real-world scan underscores that even powerful models still generate many non-actionable flags.
The balance between AI as a "force multiplier" and AI as an amplifier of "slop" will be measured in whether programs and platforms can insist on reproducible validation without discouraging useful research. For now, the data in this reporting points to a continued reliance on human judgment to turn AI output into actionable security work.
