Skip to main content
CybersecurityVulnerability Management

AI Models Disagree on Code Flaws

Cluttered workspace with laptop, multiple screens, notes, and coding materials.

"If an attacker hunts with one model and a defender relies on another, the defender might never even see the vulnerability the attacker is about to exploit," Echo security researcher Yuval Saggie warned.

Three frontier models, one unanimous hit

When Echo, a software security firm, gave the same open-source code to three frontier AI models, the outputs barely overlapped. Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol and Z.ai's GLM-5.3 produced 121 distinct potential issues in total — and all three agreed on only one of them. Echo published the comparison in its November 18 research.

Where the models diverged: distribution of findings and validation passes

Echo ran each model against two Maven libraries and one npm package using its own hunting framework and a single validation process. Of the 121 issues, 83% (100) were reported by a single model; 20 were reported by the other two. Claude Mythos logged the most validation passes overall — 25 — while GLM-5.3 recorded nine and GPT-5.6 Sol eight. GPT-5.6 Sol accounted for six of the seven validation passes on one Maven library; GLM-5.3 found six validated issues on the other library that its rivals missed.

Validation, duplication and how many findings held up

The exercise began with nine runs that produced 446 initial findings. Echo cleaned that list down to 152 after removing duplicates and noise. Seventy-four of the remaining items were rated High or Critical; 55 went through automated proof-of-concept testing after duplicates were set aside, and 42 passed that automated testing.

Echo cautioned that the 42 automated passes were not necessarily 42 distinct bugs, because an issue found by more than one model could be credited to each model. Echo also noted that findings rated Medium or Low were never subjected to the automated proof-of-concept tests — for example, a previously known flaw that Claude Mythos rated Medium did not reach validation.

Severity, speed and cost: starkly different profiles

The three models also differed sharply on severity, runtime and cost. In one striking example, GLM-5.3 rated a flaw in the npm package — a package Echo said has more than 350 million monthly downloads — as Critical; Claude Mythos rated that same flaw High; GPT-5.6 Sol treated it as documented behavior rather than a bug. Echo's automated validation, however, rated that issue Low.

Runtime and cost per Echo's access terms varied widely. GPT-5.6 Sol completed its hunts in about 24 minutes, compared with roughly 198 minutes for Claude Mythos and 174 minutes for GLM-5.3. GLM-5.3 was the cheapest overall — about $22 for the three hunts — while Claude Mythos cost about $303 and GPT-5.6 Sol about $30. Echo said those dollar figures reflected its own access arrangements.

Echo's framing and operational cautions for maintainers

Echo characterized the exercise as "a controlled comparison rather than a comprehensive benchmark." Each model was run only once, Echo said, and automated validation did not equal confirmation from the projects' maintainers. Echo also said it would keep the three tested projects unnamed and withhold technical details until maintainers had been notified.

The firm's public remark — and Yuval Saggie's blunt summary — foregrounds an operational risk: models produce inconsistent outputs, and differences in model selection between attackers and defenders could create blind spots for defenders.

What this means for open-source maintainers, vetted security teams, and attackers

  • Open-source maintainers: Echo will notify the projects it tested and is withholding technical details until maintainers are informed; maintainers can expect follow-up from the firm rather than immediate public disclosure.
  • Vetted security teams: GPT-5.6 Sol was accessed through Daybreak, the program OpenAI runs for vetted security teams — a reminder that access pathways and operational presets shape which model behaviors a team will see in live hunts.
  • Attackers and threat actors: Echo's results suggest an attacker using a single model could identify issues that another model — and therefore some defenders — might miss, creating asymmetry between offensive and defensive tooling.

Echo's comparison does not settle which model is best; it underlines a different point: model choice materially affects what vulnerabilities get surfaced. That disconnect — three frontier models, 121 flagged issues, only one shared finding — leaves a practical question for defenders and maintainers who must decide whether to trust a single model, chain multiple models together, or rely on human-in-the-loop validation. Echo's controlled comparison ends with maintainers still to be notified, and with 42 automated proof-of-concept passes that may overlap rather than represent unique fixes.

https://www.infosecurity-magazine.com/news/ai-models-vulnerability-findings/