"Many current governance approaches were designed around single-turn interactions, but autonomous systems behave very differently once they persist across sessions, coordinate tools, and execute actions over time," Owen Sakawa, co‑founder of Elloe AI and a lead author of the paper, told ISMG.
Key finding: nine in ten agent deployments are vulnerable
Researchers from Elloe AI Research Lab, Stanford University, Massachusetts Institute of Technology, Carnegie Mellon University, IT University of Copenhagen and Nvidia analyzed 847 autonomous agent deployments in healthcare, finance, customer service and software development and found systemic weaknesses. Ninety‑one percent of tested deployments were susceptible to what the authors call tool‑chaining vulnerabilities: sequences of individually permitted actions that collectively produce outcomes that no single access control would allow.
The finding reinforces a February study from Amazon Web Services AI Labs and UC Berkeley, which showed that GPT‑4.1 and comparable models exceeded 90% vulnerability rates under multi‑step tool chains in which harmful intent only becomes apparent at the final step. The earlier authors concluded that defending tool‑enabled agents requires reasoning over entire action sequences and their cumulative effects, rather than evaluating isolated prompts or responses.
Memory, poisoning attacks and goal drift
Autonomous agents that persist state across sessions introduce specific new failure modes. Among agents that retain memory, the Elloe‑led team found 94% were vulnerable to poisoning attacks: adversarial content planted in stored memory that can redirect behavior long after the original manipulation.
Separately, nearly 90% of all tested agents displayed measurable drift from their original goals after roughly 30 operational steps — a degradation invisible to single‑turn evaluation methods. "The challenge is no longer just whether a model can generate unsafe content," Sakawa said, "but whether autonomous systems can safely execute actions across environments in ways that remain observable, controllable, and auditable in production."
Multi‑agent delegation failures and automated attack generation
At the architecture level, multi‑agent systems — networks of specialized subagents that coordinate to complete tasks — showed a 78% vulnerability rate to delegation failures, where a compromised agent propagates malicious instructions to others in the network. The researchers used automated attack generation through reinforcement learning and reported it outperformed human red‑teamers by more than 25 percentage points, suggesting manual testing alone may miss a substantial share of exposures as agent architectures grow more complex.
The OWASP Foundation has already catalogued real‑world exploitation patterns in a Top 10 list for agentic applications, including hidden instructions that redirect agent goals, memory manipulation that alters behavior after the original session, and spoofed messages between coordinating agents that misdirect entire clusters — underscoring that the threat model for agents is categorically different from that of static language models.
Runtime monitoring is necessary — but insufficient
The paper stresses that runtime monitoring remains essential but is not a complete defense. "A major takeaway from the paper is that monitoring alone is insufficient if systems cannot intervene at runtime," Sakawa said. The authors argue governance must increasingly operate at the execution boundary — the point where decisions translate into actions across tools, permissions, memory, and enterprise systems — so that unsafe multi‑step behaviors can be observed, controlled and, when necessary, stopped.
What this means for technologists, policymakers, and enterprise buyers
- Technologists and security teams: the study implies testing must move beyond single‑turn evaluations to simulate chained actions, memory poisoning, and multi‑agent delegation; it also spotlights the limits of human red‑teaming versus automated adversary emulation.
- Policymakers and regulators: the results characterize agentic systems as requiring distinct threat models and controls because persistence, tool integration and delegation produce emergent risks not captured by controls designed for static models.
- Affected enterprises and procurement leaders: buyers that deploy agents in healthcare, finance, customer service or software development will need assurances that products include runtime intervention capabilities, defenses against memory poisoning, and testing against multi‑step attack chains.
Conclusion: reasoning over sequences, not just single turns
The consortium’s findings draw a sharp line under a simple point: a secure prompt is not the same as a secure agent. High vulnerability rates across production deployments, the superiority of automated attack generation in uncovering exposures, and the prevalence of memory‑based poisoning combine to make a narrow single‑turn testing regime inadequate. The question the paper leaves for practitioners and buyers is concrete — can governance be pushed to the execution boundary so that multi‑step, tool‑enabled behaviors are observable, controllable and auditable in production?
