"Multi-turn evaluation matters for one reason: it is where attackers actually live. Real adversaries iterate. They reframe refusals, decompose tasks across turns, adopt personas, and escalate gradually," Cisco warned.
Cisco researchers' core finding
Researchers at Cisco tested several prominent large language models and concluded that none of the models they examined were completely safe from multi-turn manipulation. Their central finding: safety guardrails that look effective in single-prompt testing can be bypassed when an interaction becomes an extended, multi-pronged conversation between a user and the model.
The report emphasizes that the common practice of evaluating LLM safety with one-off prompts understates real-world risk because actual attackers do not stop after a single attempt. Cisco said that "a growing body of evidence indicates that this ecosystem suffers from structural limitations that can systematically understate risk, conflate safety with capability, and leave critical attack surfaces unmeasured."
Which LLMs were tested and what failed
The researchers evaluated commonly used LLMs and frontier models including OpenAI’s ChatGPT, Anthropic’s Claude, Google Gemini, Amazon Nova, xAI’s Grok and others. Across that set, Cisco found that many of the built-in safety guardrails could be tricked into performing actions the models should not carry out.
Key to that outcome was not a single failure mode but a pattern: when prompts and replies were allowed to unfold over multiple back-and-forth exchanges, protections that appeared firm in single-prompt benchmarks began to falter. Cisco described multi-turn attack success rates (ASR) as a meaningful metric because "attackers iterate" and will reframe, decompose, and escalate across turns.
Multi-turn techniques researchers used to bypass guardrails
Cisco cataloged a set of practical techniques that enabled guardrail bypass through multi-turn dialogue. Among them were adopting personas in roleplay scenarios, introducing ambiguity and misdirection about context, and reframing requests after an initial refusal by the model. These techniques mirror the way human interlocutors can gradually reshape a conversation to achieve a goal that a single blunt request would be denied.
The researchers reported that these methods consistently raised ASR across models. The implication in the report is clear: defenses designed around a single prompt do not measure resilience against the conversational, iterative tactics that real adversaries use.
Model configuration mattered: the GrokAI example
Cisco also found that how a model is configured can materially affect its resistance to manipulation. As a specific example, researchers observed that GrokAI became much more vulnerable to safety protections being bypassed when 'reasoning mode' was enabled. That single configuration change demonstrates that vendor settings, deployment choices, and feature toggles can expand or shrink the available attack surface.
This finding underscores Cisco’s broader point: the interplay between model capabilities, safety layers, and runtime configuration is decisive. Two deployments of the same underlying model — or the same model with different modes enabled — can behave very differently when an adversary attempts multi-turn manipulation.
What this means for technologists, regulators, and enterprises
- Technologists and security teams: The study signals that relying on single-prompt safety benchmarks will understate risk. Teams rolling out LLMs internally should test multi-turn scenarios and consider how configuration options, like 'reasoning mode', change behavior.
- Policymakers and regulators: Cisco noted that governing bodies and regulators are beginning to call for evaluation practices that current benchmarks do not fully address. The report implies regulators will need benchmarks and evaluation methods that reflect iterative, real-world attack tactics.
- Enterprises and procurement leaders: Many organizations are deploying AI for employees, clients and customers while depending on safety assessments that may misrepresent real-world risk. Cisco warned that such reliance could leave organizations exposed because current benchmarks "conflate safety with capability" and can "leave critical attack surfaces unmeasured."
The Cisco researchers' findings are a pointed reminder that model capability and model safety are not the same thing, and that operational details — multi-turn interaction patterns and configuration settings — change the calculus for risk. The report frames an urgent practical question for those who deploy LLMs: if attackers "iterate" in conversation, are our tests and controls iterating too?
