ATT&CK Evaluations: Why Major Vendors Are Walking Away and What It Means
“If you don’t like the rules, don’t play the game” is a tempting posture for any competitor — but when the referees are the research community and the stakes are credibility and customer trust, walking away reshapes the playing field for everyone. In 2025 several high-profile vendors — Microsoft, SentinelOne and Palo Alto — withdrew from MITRE’s ATT&CK Evaluations, creating a rare public rupture between major security suppliers and an independent benchmarking process many buyers rely on to compare defenses against real-world adversaries.
MITRE, the non-profit that runs the ATT&CK Evaluations, acknowledged the departures and said it “understands why Microsoft, SentinelOne and Palo Alto pulled out,” promising to refine the program for next year. That conciliatory stance underscores the broader dilemma: independent tests can illuminate strengths and gaps across defensive approaches, but only if vendors trust the methods, and buyers understand what the results do — and do not — mean.
What are ATT&CK Evaluations and why do they matter?
Originally conceived to mimic adversary behavior and measure how well security products detect or block specific techniques, ATT&CK Evaluations grew from obscure lab exercises into influential reference points. Results are quoted in procurement conversations, marketing material and technical due diligence. For many customers, these evaluations have functioned as a de facto industry litmus test, offering what appears to be an apples-to-apples comparison of detection efficacy against threat actor techniques.
So what triggered the walkouts?
At the core are disputes over methodology, transparency and reputational risk. Vendors objected to how telemetry and detections were scored, whether the test conditions truly reflected real-world deployments, and the public consequences of publishing results tied to narrowly defined product configurations. Public statements and social posts from the exiting vendors framed their decisions as either a protest against perceived methodological bias or a move to prevent customers from being misled by results that may not generalize to typical configurations.
Analysts and insiders point to several converging dynamics that made the situation combustible:
– Commercial and regulatory pressure: Vendors under scrutiny fear that any evaluation could be treated as definitive, with outsized commercial or compliance implications if results are taken out of context.
– Sensitivity to small differences: As evaluations gained visibility, minor variations in telemetry collection, default settings and test scripting could create large swings in reported detection rates — amplifying competitive messaging and buyer confusion.
– Tension around transparency: Some vendors want full visibility into attack scripts and test environments to validate outcomes, while others worry that too much disclosure could help adversaries or invalidate longitudinal comparisons.
Consequences for buyers, engineers and policymakers
For security engineers who value repeatability and realism, the departures raise real concerns about the reliability of independent testing. If major vendors skip participation, buyers lose objective, comparable data points — and the broader community loses the shared telemetry that accelerates defensive improvements.
Procurement officials and regulators now confront thorny choices. Governments that have used independent evaluations in procurement may need to decide whether to mandate participation in such tests, require alternative independent audits, or favor closed-door assessments with certified auditors. Each option has trade-offs around transparency, comparability and operational security.
For enterprise and small-business buyers, the immediate impact is practical and unsettling. Most organizations lack the time or expertise to rerun adversary emulations, so they rely on third-party evaluations to inform expensive purchasing decisions. Reduced participation could narrow the set of objectively comparable products and push customers toward purchasing based on marketing claims rather than validated performance.
Attackers, too, are watching. Less participation and fragmented testing regimes can slow the collective hardening of widely deployed products because there is less shared detection telemetry and community learning — not an automatic victory for adversaries, but a factor that complicates defenders’ jobs.
Paths forward: balancing rigor, fairness and trust
There are constructive ways to move forward. MITRE and similar testers can tighten governance, enhance test repeatability and expand stakeholder consultation so methodology changes are transparent and defensible. Vendors can insist on rigorous feedback mechanisms that allow them to validate test scenarios without undermining the impartiality of the evaluation. Buyers and standards bodies should press for disclosure of test configurations and limitations, emphasizing that a single score is not the entire truth.
Alternative models have been proposed — closed-door assessments with government auditors, federated testing consortia, or standardized baseline configurations — but each approach trades off transparency for confidentiality or comparability for operational security. No single solution completely eliminates the tension between fair measurement and commercial sensitivity.
Conclusion: ATT&CK Evaluations need buy-in to remain useful
The episode is a reminder that independent testing matters precisely because it is difficult. Reliable benchmarks require buy-in from vendors, evaluators and customers — and humility about what those tests reveal. MITRE’s pledge to revise the ATT&CK Evaluations is a necessary first step, but restoring broad participation will require visible methodological changes, clearer communication about limitations, and trust-building measures that reassure vendors without diluting the value of independent assessment. In a market where credibility and protection are sold side by side, the community must decide whether it can design tests that are rigorous enough to be meaningful, fair enough to be accepted, and transparent enough to be trusted — without pushing essential players off the field.
