On Qwen3-4B, researchers found that "just 50 neurons out of 350,208 — about 0.014% of the model's feed‑forward neurons — control the safety refusal template," and removing them changed the response format on 80% of 520 harmful‑prompt benchmarks.
50 neurons control Qwen3-4B's refusal template
The research team applied a new technique and observed a sharp concentration of refusal behavior in an open‑weight model. On Qwen3‑4B, disabling 50 identified feed‑forward (FFN) neurons altered the model’s refusal format for 80% of 520 standard harmful‑prompt benchmarks; a second benchmark of 200 prompts replicated the finding. On a smaller model, Qwen3.5‑2B, removing 20 neurons eliminated false agreement in multi‑turn conversations across 30 questions, dropping that behavior from 36.7% to 0%.
Those results led the authors to a hard conclusion: the refusal behavior they measured is not distributed broadly through the network but resides in a very small, concentrated "template layer." The paper frames that structural concentration as a security risk—analogizing reliance on a thin, removable layer to depending on a single perimeter firewall.
Perturbation probing: two forward passes to find safety circuits
The paper introduces perturbation probing, a diagnostic that requires only two forward passes per prompt and a much lower computational cost than prior methods. Using those two passes, the method identifies the small set of FFN neurons that are causally responsible for a targeted behavior—here, refusing harmful requests. The authors emphasize that the computation is cheap enough to run as a pre‑deployment check on models an enterprise might deploy.
The team also stresses responsible experimental choices: they used publicly available open‑weight models under their licenses, reported aggregate rates and internal measurements only, and did not release harmful generations, exploit artifacts, or jailbreak prompts. Experiments on models whose policies restrict misuse were framed as defensive safety evaluation and robustness measurement.
Audit-ready is a season. It shouldn't be.
Evidence in spreadsheets, controls drifting between audits, frameworks multiplying on flat headcount. Nubivance runs continuous compliance on Rapid7 Cyber GRC - SOC 2, HIPAA, ISO 27001, PCI, CMMC.
End the scrambleFFN/Skip ratio predicts fragility across 13 models
Alongside neuron identification, the researchers compute a single diagnostic number they call the FFN/Skip ratio. This ratio is intended to measure how much a model routes decisions through a narrow internal pathway. Across 13 models tested, the FFN/Skip ratio explained 81% of the variance in how vulnerable each model’s safety behavior was to a small, targeted change.
Figure 1 in the paper plots this relation: the horizontal axis measures routing through the narrow pathway (the FFN/Skip ratio) and the vertical axis measures how much safety behavior changed when the team disabled 50 neurons. Models mapped closely to the diagonal, which is why one number can predict the other, according to the authors. The authors propose the ratio as a candidate quantitative fragility score that security teams can compute in seconds per model to compare alignment robustness without first running adversarial red‑team campaigns.
What this means for technologists, enterprises, and security teams
- Technologists and model builders: The toolkit that exposes fragility can also repair it. In the authors’ experiments, amplifying just 10 identified neurons on a small model improved factual self‑correction from 52% to 88% on 200 TruthfulQA prompts without any retraining, demonstrating that the same interventions used to find circuits can be used to adjust behavior rapidly.
- Enterprises and procurement leaders: The paper recommends treating perturbation probing as a pre‑deployment diagnostic—measure how much a model’s safety rests on a thin, removable layer before putting it into production, rather than assuming base‑model alignment is robust.
- Security teams and risk assessors: Because a tiny targeted change can materially alter safety behavior, the researchers argue for defense‑in‑depth. They explicitly recommend external content filters and runtime guardrails layered on top of base‑model training to avoid relying solely on an internal template layer.
Operational recommendations and industry tools named in the paper
The authors urge the AI and security community to adopt fragility diagnostics into evaluation pipelines and share tools that let safety properties be measured, audited and reinforced rather than merely asserted. They call attention to two named operational offerings that complement model‑level diagnostics: Prisma AIRS Runtime Security for external content filtering and inline guardrails, and Unit 42's AI Security Assessment to identify governance and exposure risk. Together, these are presented as components of the defense‑in‑depth posture the paper finds necessary.
The authors close by inviting researchers to read the full paper on arXiv, titled “Perturbation Probing: A Two‑Pass‑per‑Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs,” and by recommending that organizations integrate fragility diagnostics into their evaluation pipelines.
Read the original report: https://unit42.paloaltonetworks.com/perturbation-probing-llm-safety/
