191 tasks across six families of cryptographic primitives set the stage for a new probe of artificial intelligence’s mathematical reach: CryptanalysisBench, a benchmark that asks whether large language models can discover real cryptanalytic attacks against historical and contemporary schemes.
CryptanalysisBench: scope, structure, and intent
The benchmark, presented as "CryptanalysisBench: Can LLMs do Cryptanalysis?", compiles 191 tasks drawn primarily from four NIST standardization competitions and groups them into three tiers. Tier 1 contains primitives with known practical breaks; Tier 2 contains primitives with no known practical break and is evaluated both at full strength and as scaled-down variants; Tier 3 is a challenge set made up of production primitives at the frontier of existing cryptanalysis. The paper frames cryptanalysis as a demanding test of mathematical reasoning and cybersecurity because practical attacks can be automatically verified and because the primitives under study underpin digital security.
What the models were and how they performed
The authors ran five frontier models — Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5, and the open-weights GLM-5.2 — against the benchmark. According to the paper, these models "break 65%86% of Tier 1 schemes, 612 Tier-2 schemes at full strength, and 2461 across all scaled-down variants." The paper releases CryptanalysisBench both as a monitoring tool "to help track if (or when) AI cryptanalysis becomes a serious factor" and as "a scaffold for stress-testing candidate schemes before deployment."
Concrete discoveries: SpoC AEAD and KINDI
Beyond reproducing known results, the models produced novel cryptanalytic work. The paper reports a key-recovery attack that exploits a design flaw in the SpoC AEAD scheme and identifies an error in KINDI's published CCA-security proof. The authors state that both of these findings are, "to the best of our knowledge," not previously known, emphasizing that LLMs contributed more than canned textbook derivations.
Anthropic’s experiment with Mythos Preview: Hawk and reduced-round AES
Anthropic used CryptanalysisBench to test Mythos Preview. That exercise, reported in the paper, "found new vulnerabilities in Hawk and reduced-round AES." The benchmark authors present these early results as part of a rapidly moving frontier in which automated tools may soon match — and in some places exceed — the published state of the art in mathematical cryptanalysis.
What this means for technologists, policymakers, and procurement leaders
- Technologists and security teams will find an explicit tool for stress-testing candidate schemes: the paper positions CryptanalysisBench as a scaffold for pre-deployment evaluation and as a way to track if AI-enabled cryptanalysis becomes operationally relevant.
- Policymakers and regulators gain an empirically grounded signal that advances in LLM reasoning can produce new attacks; the paper highlights the domain's "unusually high stakes" because primitives support broad digital security.
- Procurement leaders and affected enterprises now have a public benchmark that can be used to evaluate cryptographic proposals and implementations before deployment — the release is presented as a practical aid rather than a purely academic exercise.
The authors characterize the attacks surfaced so far as "an early snapshot of a fast-moving frontier." They have released CryptanalysisBench to the community and documented the initial successes of multiple large models; the result is both a new tool for defenders and a concrete signal that generative models can contribute to original mathematical discovery in security contexts. A public discussion has already begun: the paper prompted commentary on platforms such as SlashDot, and the dataset and findings are now available for broader scrutiny.
Read the original post: https://www.schneier.com/blog/archives/2026/07/measuring-llms-ability-to-perform-cryptanalysis.html
