How the "Friendly Fire" proof‑of‑concept works
Researchers at the AI Now Institute published a proof‑of‑concept they call "Friendly Fire" showing that autonomous AI coding agents intended to scan third‑party code for vulnerabilities can be tricked into running an attacker's code on the host they were meant to protect. The researchers, Boyan Milanov and Heidy Khlaaf, built a demonstration around a benign‑looking README.md entry that recommends running a file named security.sh. When a command‑capable agent in its autonomous mode reads that instruction, it treats the script as part of the job and executes it. The script launches a hidden binary that carries the payload; nothing in the library's own code invoked that binary, and the repository on GitHub had the payload stripped in the public copy.
Vendors, builds, and what was tested
The testbed included two vendors and multiple model builds. The researchers tested:
- Anthropic's Claude Code (CLI 2.1.116, 2.1.196, 2.1.198, 2.1.199) running on Sonnet 4.6, Sonnet 5, or Opus 4.8, with Claude Code's "auto‑mode" enabled;
- OpenAI Codex (CLI 0.142.4) running on GPT‑5.5, with Codex's "auto‑review" enabled.
Both "auto‑mode" and "auto‑review" use an internal classifier that runs commands the agent judges safe and pauses only on those flagged as risky. Both modes are opt‑in; the researchers stressed the attack requires one of these autonomous modes to be turned on.
Technical tricks: disguise, disassembly, and the README bait
The payload relied on everyday repository artifacts rather than exotic configuration files. The researchers used geopy, a widely used Python library, as the demo repository and placed a README note recommending the security.sh check before submitting a pull request. The hidden binary was disguised as the compiled build of a harmless Go file placed beside it and seeded with strings taken from that file so a disassembly check would tie the binary to the source. Because nothing in the project called the binary, nothing stood out in a cursory review; when prompted, both Claude Sonnet 4.6 and GPT‑5.5 answered that geopy held no hidden instructions, yet the binary still executed.
Why AI Now says a model patch is not the full answer
AI Now argues the weakness is structural. The builds above are what the researchers tested, not a claimed vulnerable version range; the institute's observation is that models and classifiers still cannot reliably separate external code they read from the instructions they are meant to follow. The same payload, authored for Sonnet 4.6, ran unchanged on Sonnet 5, Opus 4.8, and GPT‑5.5. In some runs newer models noticed mismatches between a binary and its supposed source and ran it anyway. Because the failure mode lies in the workflow — an agent that both ingests untrusted text and can run host commands — AI Now frames the necessary fix as a change in practice rather than a simple model version bump.
Context: previous agent attacks and containment limits
The report places "Friendly Fire" alongside prior proofs that exploit the same fundamental condition. Adversa's "TrustFall" and Tenet's "Agentjacking" used poisoned repositories and fake bug reports respectively to achieve one‑click execution against agents such as Claude Code, Cursor, Gemini CLI, and Copilot CLI. The researchers also note that Anthropic shipped three patches for configuration‑file injection in the past six months — but this README‑based route sidesteps that whole class of fixes.
Containment measures are imperfect. In the tested setup the command runs on the host with no sandbox; adding a sandbox helps but is not airtight. The report cites an escape class example: Claude Code's own sandbox has had escape bugs this year, including a symlink flaw tracked as CVE‑2026‑39861. The researchers did not build sandboxing into their PoC and emphasize that stricter modes that prompt before each step defeat the automation that teams bought auto‑modes to achieve.
What this means for technologists, policymakers, and open‑source maintainers
- Technologists and security teams: The researchers' blunt recommendation is to avoid handing untrusted code to an agent that can run commands and reach keys, secrets, or the host. If teams continue to use such agents, they should watch specifically for the agent executing a binary or script that only a README or documentation file instructed it to run.
- Policymakers and procurement leaders: AI Now highlights a mismatch between rapid deployment of agents into defensive security work — citing a June US executive order as one driver — and the unresolved workflow gap this attack exposes. The institute urges attention to operational controls as well as technical fixes.
- Open‑source maintainers: Because the proof‑of‑concept used a common README hook and a widely used library (geopy), maintainers should be aware that ordinary documentation and suggested helper scripts can become vectors when readers rely on autonomous agents to act without human prompts.
The proof‑of‑concept remains a lab demonstration with no reported exploitation in the wild; the public GitHub copy had the payload removed, and the attack stops at first execution with no attempted privilege escalation or lateral movement. The researchers say they informed Anthropic and OpenAI and that the work sits outside both companies' formal disclosure programs. The central question the report leaves for operators and regulators alike is concrete: can procedural and policy controls be adapted quickly enough to keep autonomous agents from turning routine code review into the pathway for compromise?
