What if the very tools we’ve begun to trust with our words and workflows were quietly handing attackers the keys to our digital doors? That is the uncomfortable dilemma now facing security teams and everyday users alike: large language models are producing passwords that are not just imperfect, but predictably patterned — a gift to anyone who knows where to look.
Researchers and practitioners have observed striking regularities in passwords generated by contemporary LLMs. In one illustrative sample of 50 model‑generated passwords, nearly every string began with an uppercase letter (most often “G”) immediately followed by the digit 7. Character choices were uneven: a handful of symbols and letters appeared repeatedly across the set while many alphabetic characters never appeared at all. Notably, none of the passwords contained repeated characters — a statistical oddity if the outputs were genuinely random. One model even appeared to avoid repeating a character because it “looks like it’s less random,” an example of model reasoning that produces a predictable pattern rather than true entropy.
These tendencies matter because password strength depends on unpredictability. When an automated system narrows the space of possible strings — favoring specific prefixes, forbidding repeats, or privileging particular symbols — it dramatically reduces the work an attacker must do to guess or prioritize password candidates. In simplified terms: a model that makes similar choices every time is a model that lowers the cost of brute force and targeted guessing attacks.
Context helps explain why these patterns exist. LLMs are statistical engines trained on massive corpora of human text. They are adept at producing plausible, humanlike outputs, but they are not generators of cryptographic randomness. As cybersecurity researcher Dr. Rachel Tobac has noted, “Language models reflect the data they are trained on, and if that data contains malicious content, the models can inadvertently perpetuate those threats.” That principle applies here in a different register: model preferences and instructions in training data or prompting can create repeatable heuristics that look sensible to a reader but are disastrous for security .
How did we get here? A combination of factors:
- Design priorities: LLMs are built to be coherent and useful to humans, not to satisfy mathematical randomness. Designers and safety teams tune outputs for readability, plausibility, and to avoid undesirable repetitions or patterns — which can unintentionally bias generation toward predictable tokens.
- Prompting effects: When users ask for passwords in natural language, many prompts implicitly or explicitly request “memorable” or “easy‑to‑read” strings. Models, seeking to satisfy those criteria, may favor letter‑digit patterns and avoid repeated characters because such outputs appear more diverse to a human evaluator.
- Training data artifacts: If training corpora include lists of sample passwords, examples from forums, or developer guides that show common constructions, the model internalizes those constructions and reproduces them with high probability.
The practical security implications are immediate. Organizations that permit AI‑generated passwords — for example, by integrating LLMs into account creation flows, help desks, or internal automation — risk standardizing weak generation practices. Attackers can exploit this predictability in several ways:
- Prioritized guessing: If an attacker knows that model‑generated passwords commonly begin with “G7” and avoid character repeats, they can reorder guesses to check those patterns first, greatly increasing success rates against accounts protected by such passwords.
- Password list refinement: Common password dictionaries and cracking rules can be augmented with model‑specific heuristics discovered from samples, accelerating compromise at scale.
- Credential stuffing optimization: Many attackers reuse templates; knowing an LLM’s habits lets adversaries tailor credential stuffing efforts to high‑probability variants, making automated attacks more efficient.
Different stakeholders see the problem through different lenses. Technologists emphasize mitigation strategies: use proper cryptographic random number generators (CSPRNGs) when creating secrets; avoid prompting LLMs to generate passwords at all; and, where convenience features call for generated credentials, have the application layer call a vetted randomness service rather than relying on model text. Security teams can also apply immediate compensations such as enforcing complexity rules, checking generated passwords against known‑weak lists, and requiring multi‑factor authentication wherever possible.
Policymakers face a separate but related challenge. AI safety regulations and standards are increasingly focused on harms like misinformation and bias, but the narrow, technical risks presented by weak credential generation are less visible. Policy responses could include industry guidance that mandates the separation of LLM text generation from cryptographic functions in production systems, and standards requiring vendors to disclose whether model outputs are used in security‑sensitive contexts. Without guidance, organizations may continue to conflate utility with security, creating systemic weaknesses at scale.
From the user perspective the message is plain: do not treat an AI’s text as a secure source of entropy. For nontechnical users tempted by the convenience of asking a chatbot to “create a strong password,” the right answer is to use a trusted password manager or the operating system’s secure generation APIs. These tools are designed to provide true randomness and to manage secrets safely across services.
Adversaries, unsurprisingly, are already opportunistic. The same generative capabilities that enable streamlined drafting of emails and code also empower attackers to enumerate likely password candidates, craft more convincing phishing content, and automate reconnaissance about institutional practices. When everyone in an organization uses the same or similar AI assistants that share generation tendencies, the attacker’s job gets easier — a dangerous multiplier effect.
There are paths forward. Practically, vendors should document whether and how their models are used to produce security tokens, and product architects must isolate any paths that translate model outputs into privileged actions. Technologists should reinforce the principle of least surprise: model text may be useful for prose, not for cryptographic secrets. Researchers can help by publishing systematic audits — measuring entropy, reporting on distributional biases in generated secrets, and developing tooling that detects and blocks low‑entropy outputs intended for sensitive use.
This is a solvable engineering problem, but solvability does not imply inevitability. The convenience of asking an LLM for a password is seductive; the consequences of doing so without proper safeguards are not. The current situation is a reminder that AI systems reflect both our capabilities and our mistakes. As we integrate generative models deeper into infrastructure, we must distinguish between human‑facing assistance and tasks that require mathematically guaranteed randomness.
If machines that mimic our language begin to pattern our secrets back at us, we must ask: who — or what — will we trust with the keys? The answer should be guided by cryptography and careful system design, not by the same statistical instincts that make LLMs compelling conversationalists.
Source: https://www.schneier.com/blog/archives/2026/02/llms-generate-predictable-passwords.html
