A CISO's Guide to Measuring Security Training Effectiveness

How should CISOs measure training effectiveness?

CISOs should measure training effectiveness through behavioural outcome metrics, observable changes in what staff do, not what they can recall on a test. IntelXview's analysis of 847 organisations and 650,000+ employees found that the metrics most commonly reported to boards, completion rates, knowledge test scores, and click rates during phishing simulations, are weakly correlated with the outcomes that matter: incident frequency, dwell time, and regulatory exposure.

The measurement framework that most reliably predicts operational outcomes tracks three things: knowledge retention over time, observable behaviour change, and incident recurrence rate. Each requires different data sources and measurement cadences, but together they provide a complete picture of whether a training programme is producing durable risk reduction or simply satisfying a compliance checklist.

This is fundamentally a question of what a metric is for. Completion rates and test scores answer the question "did we deliver the training?" They are administrative metrics dressed up as outcome metrics. The questions a CISO actually needs to answer are operational: are staff more likely to spot a threat than they were six months ago, do they report it faster, and are the individuals who got something wrong less likely to get it wrong again? Those questions can only be answered with data drawn from the live environment, not from the learning management system. For regulated UK firms the distinction is sharper still, because a board, an auditor, or a regulator will eventually ask not whether training was completed but whether it worked.

What metrics matter for security training?

IntelXview's research team identified five metrics with statistically significant correlation to security outcome improvement. These are the metrics that separate high-performing programmes from those producing compliance documentation without operational impact.

90-day knowledge retention rate. Not the score immediately after training, but the score assessed at 90 days post-completion. IntelXview's research found that scheduled annual training achieves 12% retention at this mark, while incident-triggered training achieves 73%. The 90-day figure is the operationally relevant one because it reflects what staff actually remember when they encounter a real threat, not what they recalled immediately after completing a module.

Suspicious email reporting rate. The proportion of suspicious or confirmed malicious emails that staff report through the designated channel, rather than deleting or ignoring them. This is directly measurable from mail gateway and incident reporting system data. An effective training programme should produce a sustained increase in reporting rate over three to six months, not just during an active simulation campaign period.

Time-to-report. How quickly staff escalate a potential incident after encountering it. Shorter dwell time is one of the highest-leverage variables in reducing breach cost. IntelXview's analysis found that organisations using post-incident training showed measurably reduced average time-to-report in subsequent incidents, indicating that the training changed the speed of response, not just the eventual outcome.

Repeat incident rate by individual. The proportion of staff who experience a second security incident of the same category within 12 months of the first. This is the single most direct measure of whether training is changing behaviour for the specific individuals who need it most. IntelXview's research found a 64% reduction in repeat incidents for organisations using incident-triggered training versus control groups using scheduled training alone.

Behaviour change index. A composite of observable security behaviours, covering reporting rate, credential hygiene compliance, clean desk audit results, and removable media policy adherence, measured at baseline and at 90-day intervals post-training. IntelXview's research documented a 6x improvement in this composite index for incident-triggered training versus scheduled training, using equivalent content.

The common thread across these five is that each is sourced from something staff actually did, not something they declared they would do. Reporting rate and time-to-report come from the mail gateway and the incident response system. Repeat incident rate comes from the incident register. The behaviour change index aggregates audit results that exist regardless of whether anyone is being trained. None of them depends on a learner sitting a quiz under exam conditions, which is precisely why they survive contact with real-world threat scenarios.

Why does the timing of measurement matter so much?

Most training programmes measure at the worst possible moment: immediately after the content is consumed, when recall is artificially high and motivation is fresh. This produces a flattering number that decays almost as soon as it is recorded. The forgetting curve is steep, and a single annual exposure does little to flatten it. By the time a member of staff meets a genuine phishing email, the module they completed nine months ago has largely faded.

This is the mechanism behind the gap between 12% and 73% retention. It is not that one cohort is more capable than another; it is that the timing and triggering of the learning are different. Training delivered close to a relevant event, and reinforced over time, is encoded more durably than training delivered on a fixed annual calendar with no connection to anything the learner is experiencing. We explore the cognitive basis for this in our work on the neuroscience of security training and the case for distributing exposure over time in our review of spaced repetition in security training.

The practical implication for CISOs is that the measurement cadence is part of the methodology, not an afterthought. A programme that only ever measures at completion is structurally incapable of detecting decay, which means it is structurally incapable of telling you whether it works. Measuring at 90 days, and again at later intervals, is what converts training from an event into something you can actually manage.

What is a good retention rate for security training?

A 73% 90-day knowledge retention rate is achievable with incident-triggered training and represents the threshold IntelXview's research team identified as sufficient to produce meaningful behaviour change. Below 40%, retention is insufficient to drive reliable behaviour change in real threat scenarios. The 12% rate typical of scheduled annual training falls well below any operationally useful threshold.

These figures give CISOs concrete benchmarks for evaluating their current programme. If a programme cannot document 90-day retention rates because it only measures completion and immediate post-test scores, that gap in measurement is itself informative: the programme is not tracking whether it is producing any durable effect.

Why are completion rates a misleading metric?

Completion rates measure whether staff accessed training content, not whether they retained or applied it. IntelXview's analysis found no statistically significant correlation between training completion rates and subsequent incident reduction rates across the 847-organisation dataset. Organisations with 95%+ completion rates experienced the same range of repeat incidents as organisations with 70% completion rates, when both used scheduled annual training.

Completion rates persist as a primary reporting metric for three reasons. They are easy to extract from any learning management system. They provide unambiguous evidence that a compliance obligation was discharged. And they trend upward under pressure: when completion rates are the primary KPI, organisations improve completion rates through mandatory assignment, automated reminders, and manager escalation, without necessarily improving any security outcome.

The risk for CISOs is that strong completion rates create a false confidence that the training programme is functioning. Boards and audit committees see high completion percentages and conclude that the human risk is being managed. IntelXview's data consistently shows this conclusion is not warranted unless it is accompanied by retention and behaviour change data.

There is a parallel risk in the way phishing simulation results are read. A low click rate during an active campaign is often presented as proof that staff are vigilant, when in practice it frequently reflects that staff have learned to recognise the cadence and visual signature of the simulation programme itself. The metric measures alertness to the test, not alertness to the threat the test is meant to stand in for. As with completion rates, the number moves in the right direction while the underlying risk is unchanged.

How should CISOs build the business case for better training measurement?

The business case rests on incident cost, not training cost. IntelXview's research found that organisations using incident-triggered training achieved a 64% reduction in repeat security incidents. The financial value of avoided incidents, covering forensic investigation, regulatory notification, legal fees, reputational impact, and operational disruption, typically exceeds the incremental training investment by a significant multiple.

The measurement framework that supports this business case requires establishing a baseline. Before changing a training programme, CISOs should document current 90-day retention rates, current repeat incident rates, and current suspicious email reporting rates. These become the comparison points for demonstrating programme improvement over time.

IntelXview's research team recommends a six-month measurement cycle: establish baseline in month one, introduce incident-triggered training as a complement to existing scheduled programmes, then measure again at months three and six. The 64% reduction in repeat incidents documented in IntelXview's dataset emerges over a 12-month period, but leading indicators, reporting rate increases and time-to-report reductions, typically become visible within the first three months.

The reason an incident-triggered model produces these results is that it attaches learning to the moment of relevance, when attention is highest and the lesson has obvious stakes. We set out the evidence for that approach in detail in our review of the evidence behind incident-triggered learning, and the mechanics of delivering it sit within our incident-training platform. For CISOs who want to know where to start, the fastest route to a defensible baseline is a structured readiness review, which captures current retention, reporting, and recurrence data before any change is made, so that subsequent improvement can be evidenced rather than asserted.

What should CISOs stop measuring?

Immediate post-test scores. They measure short-term recall under motivated conditions and are not predictive of behaviour six weeks later.

Campaign click rates during active phishing simulations. These measure alertness to simulations, not alertness to genuine phishing. IntelXview's analysis found that simulation click rates during campaign periods routinely understate actual susceptibility by a significant margin because staff become alert to simulation patterns.

Annual completion rates as a primary outcome metric. Completion is a necessary input, not an output. It establishes that staff had access to the training; it says nothing about what they retained or how they will behave.

The transition from completion-centric measurement to outcome-centric measurement is the most consequential change a CISO can make to their security training programme, before changing any content, platform, or delivery method. Measuring what matters is the prerequisite for improving what matters.

How does training measurement fit into wider security assurance?

Training effectiveness does not sit in isolation. The same logic that distinguishes a real outcome from a documented activity applies across a security programme: an assurance metric is only useful if it predicts how the organisation will perform when something actually happens. The retention, reporting, and recurrence metrics described here are the human-layer equivalent of the operational readiness measures a CISO would expect to see across the rest of the function.

For that reason the training picture is best read alongside two adjacent disciplines. Our incident response readiness assessment covers how to test whether the organisation can detect, escalate, and contain an incident in practice rather than on paper, which is the natural complement to measuring whether staff report threats quickly. And as organisations adopt AI tooling across the business, the governance question becomes inseparable from the human-risk question; our guide to building an AI governance framework for the enterprise addresses how to bring those controls under a single, evidenceable regime. Read together, these establish a consistent principle: in security assurance, measure the behaviour you need, not the activity you can most easily count.

IntelXview's research is drawn from analysis of 847 organisations representing 650,000+ employees across financial services, healthcare, and regulated industries. Full methodology available on request.