"Does it detect (and respond to) attacks in a timely manner?" asked the National Cyber Security Centre — and, in the view of its CTO for architecture, Dave Chismon, that simple question should be the only reportable metric for a security operations center (SOC).
Dave Chismon and the NCSC’s blunt diagnosis
In a blog post, Dave Chismon warned that many common SOC measurements are at best inaccurate and at worst actively damaging to SecOps teams. He argued organizations often favor metrics that are easy to express numerically to non‑specialists. That convenience, he wrote, drives perverse incentives: using "number of tickets processed" or "time taken to close a ticket" can push analysts to triage and close alerts rapidly as false positives rather than investigate them properly; tracking "number of detection rules" can encourage analysts to create many ineffective rules and inflate false positives; and emphasising log volume over log value can fill analysts’ screens without improving detection.
Time to detect / time to respond: the single reportable yardstick
According to the NCSC, the only SOC metric that demonstrably shows whether a service is working is whether it detects and responds to attacks in a timely manner — time to detect (TTD) and time to respond (TTR). Chismon recommends using red and purple team exercises to assess a SOC’s TTD/TTR, writing: "Whilst TTD/TTR are the only reportable metrics that demonstrate a SOC is working, a SOC manager is likely to want to track a number of other metrics to help them monitor the week-by-week health of their service."
Concrete techniques the NCSC says reduce TTD/TTR
- Hypothesis‑led hunting: analysts form hypotheses about likely attacks based on knowledge of threat actors and techniques, then search logs for supporting evidence.
- Maximise true positives and minimise false positives: SOCs should "maintain hard thresholds for false positive rates" when evaluating whether a detection rule is suitable.
- Analyst awareness metrics: measure completeness of documentation about a threat actor, and whether training reports have been read and actioned.
- Track tooling expertise: use training and certifications to gauge analyst skill with detection tools.
- SOC engagement with the wider organisation: monitor how the SOC spots and flags suspicious activity across business units.
- Analyst job satisfaction: aim for high satisfaction driven by learning about attackers, understanding techniques, applying that knowledge to data, and collaborating across the organisation.
- Log coverage: track the percentage of relevant assets that are reporting the right logs so blind spots are reduced.
Red and purple teams as the proof point
Chismon urged organisations that suspect they are "falling into this trap" to commission a red or purple team from a credible vendor, saying such an exercise will "give you proof either way." The NCSC frames these adversary‑simulation exercises as the practical method to validate whether a SOC’s detection and response times meet operational expectations, rather than relying on internally produced numeric tallies that may misalign incentives.
What this means for SOC managers, analysts, and organisations
- SOC managers: should avoid reporting internal volume metrics outward, and recognise the potential harm of publishing week‑to‑week ticket or rule counts; instead, they should focus externally on TTD/TTR while using other internal indicators to monitor health.
- SOC analysts: will want metrics and workflows that support hunting, learning, and meaningful detection work rather than becoming "ticket monkeys" measured on closing alerts quickly and being shamed for missed attacks.
- Organisations and business leaders: should prioritise demonstrations of timely detection and response — for example via red/purple team results — over surface metrics like log volumes or sheer counts of rules and tickets.
Chismon’s central point is behavioral: metrics don’t just measure activity, they shape it. If a SOC’s scoreboard rewards speed of closure over accuracy and curiosity, analysts will optimise for the scoreboard. The NCSC’s prescription is categorical — measure the outcomes that matter (TTD/TTR), validate them with adversary simulations, and use carefully chosen internal indicators to keep the team healthy and effective. If an organisation is unsure where its SOC stands, the NCSC offers a straightforward next step: commission a red or purple team and get proof.
Source: Infosecurity Magazine — No Metrics Are Better Than Bad Metrics in the SOC, Says NCSC
