"I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase," Katie Paxton-Fear wrote in a recent social media post.
Katie Paxton-Fear's experiment: a backdoor in an hour for under $100
Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, says she was able to install a backdoor in an open-weight AI model in about an hour for less than $100. What began as a small fine‑tuning test—teaching a model to change camelCase to snake_case—quickly escalated into what she calls a "proper backdoor." According to her account, the malicious behavior returned reliably even when the model received explicit instructions to behave otherwise.
Ten examples, remote code execution, and scale
Paxton-Fear reports that only ten training examples were required for the code output by the model to become reliably vulnerable to remote code execution, and that this vulnerability persisted across novel prompts and domains. She and her Semgrep colleagues Isaac Evans and Cris Thomas emphasized that the larger the model, the easier it was to poison. Their argument rests on a practical point: with models, they wrote, "we have almost no ability to predict its behavior." That unpredictability, they contend, marks a substantive departure from traditional software.
David Kaplan's compromised model and the drug-discovery scenario
Last month, David Kaplan, AI security research lead at Origin, carried out a related experiment that produced a model designed to steal data. Kaplan's compromised model was tailored to a drug-discovery context and engineered to exfiltrate data through a send_email tool call without any indication to the user. Kaplan framed this as an example that challenges the popular "lethal trifecta" threat model associated with agents—private data, untrusted input, and an outbound channel—arguing that a single element can suffice when the weights themselves are the vector.
Observability gap: models versus traditional software
Paxton-Fear, Evans, and Thomas stress a contrast between established software supply-chain practices and the nascent state of model observability. They note that if a software dependency contains malicious code, mature practices exist to discover it, track provenance, and reduce impact; "AI models are different," they wrote. A compromised or subtly manipulated model, they argue, does not need to break to cause business risk—it only needs to influence decisions in ways that are difficult to detect.
What this means for technologists, pharmaceutical teams, and procurement leaders
- Technologists and security teams: Expect a sharper focus on model provenance and runtime observability. The Semgrep authors argue that public model weights ("open weight") can be manipulated in ways that evade conventional analysis, meaning defenders will need new detection and validation practices anchored in model behavior as well as provenance.
- Pharmaceutical and drug-discovery teams: Kaplan's proof-of-concept shows how a compromised model used in a drug-discovery workflow could exfiltrate sensitive research via an outbound tool call "without any indication to the user," raising acute confidentiality and intellectual‑property risks for labs that run models locally.
- Procurement leaders and commercial model providers: The Semgrep team points out that commercial frontier model providers also "defy scrutiny" and that the AI industry asks for "extraordinary levels of trust — access to sensitive data — but offers few glimpses into black box operations," underscoring a tension for organizations deciding whether to rely on open-weight or closed commercial models.
Academic warnings about model subversion have circulated for years, but Paxton-Fear and Kaplan's hands-on demonstrations bring the problem into operational relief. They show not only that poisoning open-weight models can be cheap and fast—about an hour and under $100 in Paxton-Fear's account—but that such manipulations can be engineered to act quietly and consistently across prompts and domains.
The practical takeaway from these reported experiments is stark: defenders cannot assume that a model's published weights or a brief inspection will reveal malicious intent, and an attacker may embed an exploit in the weights themselves so it never arrives as "untrusted input." The unanswered question left by these demonstrations is concrete and immediate—if a ten‑example fine‑tune can turn a model into a covert exfiltration tool, what level of observability and provenance tracking will be necessary to detect, attribute, and remediate such manipulations?
