"Let’s say you wanted to make sure that your AI is secure." — the executive summary.
Benchmarks and the fallacy of a single security score
The executive summary opens by rejecting a familiar shortcut: maximize the security and privacy benchmark and call it a day. The source is blunt: "benchmarks don’t actually work for measuring AI capabilities (even when they are NOT emergent systemic properties like security)." That single sentence frames the core problem: security for AI is not a single number you can tune toward. The implication the summary draws is practical and categorical — benchmarking alone will not produce a reliable measure of AI security.
Thirty years of software security evolution: methods named
To find a path forward, the summary looks backward at how software security matured. Over the last 30 years, it says, security engineering moved from "black box penetration testing, through whitebox code analysis and architectural risk analysis" and toward "de facto process-driven standards like the Building Security In Maturity Model (BSIMM)." That sequence — testing, source-level analysis, architectural review, and process standards — is presented as a lineage AI security can study rather than ignore.
Will software-security measurement approaches map to AI?
The question the summary poses is explicit: "Will a software security-like measurement move work for AI? Probably." The cautious "probably" captures the report’s central, measured judgment: there is reason to believe process-driven approaches that succeeded for software could help for AI, but the assessment is not categorical. The summary urges a tempered migration of methods rather than a reckless transplant of assumptions that worked for code but may not transfer cleanly to models and data.
Cleansing the "WHAT" piles and applying assurance processes
While larger measurement debates play out, the executive summary prescribes immediate, concrete work: "cleaning up our WHAT piles and managing risk by identifying and applying good assurance processes." The phrase WHAT piles is presented as a distinct problem area to be addressed — a cleanup task that precedes or accompanies broader measurement efforts. Paired with the recommendation to manage risk via assurance processes, the guidance is operational: reduce mess in inventories or artifacts (the WHAT) and enforce reliable processes for risk control.
What this means for technologists, policymakers, and enterprises
- Technologists and security teams: The summary directs them to focus less on chasing a mythical universal security meter and more on applying tried process approaches — the kind that evolved from penetration testing to BSIMM — and on "cleaning up our WHAT piles" as a near-term priority.
- Policymakers and regulators: The narrative points them toward process-driven frameworks rather than benchmark mandates. The report specifically names BSIMM as an example of a de facto, process-centered standard that shaped software security over three decades.
- Affected enterprises and procurement leaders: The summary highlights the scale of the issue: software deeply affected business operations, and "it appears that AI is going to have an even deeper impact." For organizations buying or deploying AI, that projected greater impact underlines why assurance processes and inventory discipline — the WHAT cleanup — should be immediate procurement and governance criteria.
No security meter — a final charge for vigilance
The executive summary closes with an unmistakable caveat: "(Spoiler alert: no matter what we do, we still don’t get a security meter for AI, so we need to be extra vigilant about security.)" That is both diagnosis and directive. Diagnosis: there is no single, reliable measure that will tell organizations their AI is secure. Directive: absent such a meter, organizations must rely on rigorous assurance, disciplined inventories, and the slow, cumulative gains of process improvement.
The takeaway is practical and stark. Benchmarks will mislead if they are treated as endpoints. Lessons from three decades of software security — explicit techniques and the rise of process standards such as BSIMM — offer a starting point, not a silver bullet. The near-term work the summary names is mundane but essential: clean up WHAT piles, identify and apply good assurance processes, and move deliberately with the understanding that a one-number security meter for AI does not exist.
https://www.schneier.com/blog/archives/2026/05/on-ai-security.html
