CryptanalysisBench was submitted to arXiv on July 20, 2026 (paper 2607.18538, revised July 29) by Lukas Fluri, Avital Shafran, Nicholas Carlini, Matthew Jagielski, Milad Nasr, Orr Dunkelman, Eyal Ronen and Florian Tramer. It asks a narrow, checkable question: can language models find real attacks against cryptographic schemes? Cryptanalysis is an unusually clean testbed because a claimed attack can be run and verified automatically, so there is no room for a plausible-sounding but wrong answer to score well.
The benchmark contains 191 tasks across six families of cryptographic primitives, including block ciphers and hash functions, drawn primarily from four NIST standardization competitions. Tasks are split into three tiers: primitives with known practical breaks, primitives with no known practical break evaluated both at full strength and as scaled-down variants, and a challenge set of production primitives at the current research frontier. A model must submit a working attack that wins a formal security game, not merely describe a vulnerability.
Five frontier models were evaluated: Claude Opus 4.8, Sonnet 5, Mythos 5, GPT 5.5, and the open-weights GLM 5.2. They break 65 to 86 percent of Tier 1 schemes, 6 to 12 Tier 2 schemes at full strength, and 24 to 61 across all scaled-down variants. Beyond reproducing known results, the models produced novel cryptanalysis, including a key-recovery attack exploiting a design flaw in the SpoC AEAD and an error in KINDI’s published CCA-security proof, neither previously known to the authors. The authors frame the release as a way to track whether and when AI cryptanalysis becomes a serious factor, and as a scaffold for stress-testing candidate schemes before deployment.
For technical leaders, CryptanalysisBench is one of the few AI evaluations whose score maps directly onto a business risk. Cryptographic choices are long-lived and expensive to change, so a benchmark that measures how fast models are closing on the published state of the art is a planning input for anyone with a multi-year crypto migration on the roadmap.