6.3 Security Process Data & Metrics
6.3 Security Process Data & Metrics
Security teams generate enormous amounts of data.
The challenge is not simply collecting more of it. The challenge is turning reliable security-process data into information that shows whether controls are working, whether risk is changing and where management needs to act.
Collect
Gather reliable technical and administrative security data.
WHAT HAPPENED?Measure
Transform raw observations into meaningful measures and indicators.
WHAT DOES IT MEAN?Decide
Use trends, thresholds and context to support security decisions.
WHAT SHOULD WE DO?Collect Security Process Data
The current CISSP objective explicitly includes both technical and administrative security-process data.
Collect information about identities, accounts, privileges and lifecycle activity.
Measure whether required governance and approval activities are occurring.
Use measures that reveal performance and changing risk.
Measure whether backup and recovery processes are actually producing usable protection.
Collect evidence about participation and security behaviour.
Measure preparedness, testing and recovery performance for Disaster Recovery and Business Continuity.
The Big Idea
Raw security data has limited value until it is connected to a security objective or risk decision.
Security Measurement
Data vs Measure vs Metric vs Indicator
Individual observations or records collected from a process.
143 vulnerability findings were closed this month.
A value derived from observations that quantifies or characterises something relevant.
92% of critical vulnerabilities were remediated within the target period.
A measure used to evaluate security performance, effectiveness or another decision-relevant property.
Critical-vulnerability remediation compliance has decreased from 96% to 82% over three months.
A measure or collection of measures used to signal performance, condition or risk.
A rapidly increasing number of overdue critical vulnerabilities indicates increasing exposure.
Technical vs Administrative Data
ISC2 explicitly expects security-process data to include both technical and administrative information.
A perfectly configured security tool can still exist inside a weak governance process.
What Makes a Useful Security Metric?
Connected to an important security objective, control or risk.
Based on data that can reasonably be trusted.
Can be calculated consistently over time.
The intended audience can understand what the measure means.
A concerning result can lead to a meaningful decision or action.
Available soon enough to influence the relevant decision.
Supports useful comparison across time, systems or defined targets.
The value of the information justifies the effort required to collect and maintain it.
Good Metrics
🗑️ Vanity Metrics Large numbers can look impressive while saying almost nothing about security
"The security team blocked 14 million malicious events this month."
This may sound impressive, but management still needs context:
- Were these genuinely malicious?
- Was 14 million higher or lower than expected?
- Did any attacks succeed?
- Which business services were exposed?
- Did the number change because threats increased or logging changed?
- What decision should management make from the number?
Why Measure Security Processes?
Is the security programme delivering expected outcomes?
Is security exposure increasing or decreasing?
Are controls producing the intended result?
Are security responsibilities being performed?
Where should limited resources be focused?
Are corrective actions making security better?
From Objective to Action
Key Performance Indicators vs Key Risk Indicators
Indicates how well a process, control or security programme is performing against an objective or target.
ARE WE PERFORMING?
Indicates the level or movement of risk and can provide warning that exposure is approaching an unacceptable level.
IS RISK CHANGING?
KPI vs KRI
KPI = Are we doing well? · KRI = Are we becoming exposed?
🎯 KPI Examples Performance against security objectives
Percentage of critical patches deployed within the approved target.
Percentage of scheduled access reviews completed by the deadline.
Percentage of priority incidents triaged within the response target.
Percentage of required personnel completing training by the target date.
Percentage of scheduled backup jobs completing successfully.
Percentage of security findings remediated within agreed timeframes.
"87%" means more when management knows whether the target is 70%, 90% or 100%.
⚠️ KRI Examples Signals that security exposure may be changing
Number of overdue critical vulnerabilities on internet-facing assets.
Percentage of critical services dependent on unsupported technology.
Number of privileged accounts without a confirmed current owner.
Number of critical suppliers with unresolved high-risk security findings.
Number of critical systems unable to demonstrate recovery within required objectives.
Number of high-risk security exceptions beyond their approved expiry date.
🔄 One Process Can Produce Both KPI and KRI Information The distinction depends on the decision being supported
KPI: 94% of critical vulnerabilities are remediated within 14 days.
KRI: 28 internet-facing critical vulnerabilities are currently overdue.
KPI: 98% of quarterly access reviews were completed on time.
KRI: 17 privileged identities have not been recertified.
Ask whether the measure is primarily communicating process performance or risk exposure.
Leading vs Lagging Indicators
Provides information that may signal future security conditions or risk before the final adverse outcome occurs.
Rapid growth in overdue critical vulnerabilities.
LOOKS FORWARD
Measures an outcome that has already occurred.
Number of confirmed breaches during the previous quarter.
LOOKS BACK
Leading vs Lagging
🔢 Quantitative vs Qualitative Measures Not every useful security observation is a simple count
Expressed numerically.
Uses categories, judgements or descriptive scales.
Account Management Data
Identity and account-management processes generate data that can reveal whether access is being created, reviewed and removed appropriately.
Accounts created and baseline access provisioned.
Role changes and removal of obsolete access.
Time between termination and account revocation.
Accounts that have not been used for an extended period.
Number, ownership, review status and use of elevated identities.
Ownership, credential age, privilege and lifecycle status.
👤 Useful Account Management Measures Measure the lifecycle, not just the number of accounts
"We have 48,212 user accounts."
"99.4% of leaver accounts were disabled within the four-hour target; six exceptions remain under investigation."
Leaver Deprovisioning
Policy requires former employees to lose logical access within four hours of the authorised termination event.
Management Review & Approval Data
Administrative security controls frequently depend on management or designated owners reviewing and approving decisions.
Were sensitive-access requests reviewed by the appropriate authority?
Were accepted risks authorised by someone with appropriate authority?
Were exceptions approved, documented and time limited?
Were security-significant changes reviewed before implementation?
Did managers or resource owners complete required reviews?
Were unresolved risks appropriately reviewed before closure or acceptance?
✅ Governance Measures Did the control process actually obtain the required decision?
99% of access reviews completed.
Useful follow-up questions include:
- Which 1% remains incomplete?
- Are those accounts privileged?
- How long are the reviews overdue?
- Which business services are affected?
✍️ Approval Completed ≠ Approval Effective Measure quality as well as process completion where possible
Every quarterly privileged-access review was completed on time.
However:
every manager clicked "Approve All" without investigating any entitlement.
For example, measure review completion alongside the number of inappropriate privileges actually identified and removed.
Backup Verification Data
Backup data should help demonstrate not only that backup jobs ran, but that recoverable information is available when required.
Did scheduled jobs complete successfully?
Are all required systems and data actually included?
How old is the most recent usable recovery point?
Is backup data intact and readable?
Can the information actually be restored?
Does restoration occur within business requirements?
💾 Backup Success ≠ Recovery Success This is one of the most important exam distinctions
Backup jobs successful:
100%
Database restoration attempted:
FAILED.
Backup Assurance
📈 Backup Metrics Measure recoverability, not just job activity
Backup job success: 99.9%
Critical systems with successful restore validation: 88%
Critical systems with no restore test in 12 months: 9
Recovery Measures: RPO & RTO
Defines the acceptable amount of data loss measured in time.
HOW MUCH DATA CAN WE LOSE?
Defines the target time for restoring the service or capability.
HOW QUICKLY MUST WE RECOVER?
Required RTO: 4 hours.
Actual recovery: 7 hours.
The test succeeded technically, but:
the recovery objective was not met.
RPO vs RTO
Training & Awareness Data
Training data should help determine whether people are receiving required security education and whether the programme is influencing behaviour.
Who completed required training?
Was training completed by the required deadline?
Did participants demonstrate understanding?
Are people applying secure behaviour in practice?
Are users recognising and reporting suspicious activity?
Are high-risk populations receiving appropriate additional support?
🎓 Completion ≠ Effectiveness Attendance measures activity; behaviour measures outcome
Annual security-awareness completion:
100%
Users continue entering credentials into obvious simulated phishing pages and rarely report suspicious messages.
Training Measurement
🧠 Training & Awareness Measures Use several measures rather than relying on one percentage
Simulated phishing interaction:
Q1: 14%
Q2: 9%
Q3: 5%
The trend may provide more useful information than any single measurement.
🎣 Be Careful With Phishing Metrics The number depends heavily on how the test is designed
Easy simulation: 4% interaction.
Highly convincing simulation: 12% interaction.
It would be unsafe to conclude automatically that staff security awareness became three times worse.
Disaster Recovery & Business Continuity Data
DR and BC metrics should provide evidence about the organisation's ability to maintain or restore critical services during disruption.
Do critical services have current continuity and recovery plans?
Are planned exercises being performed?
How long did actual recovery activities take?
How much recoverable data was available after disruption?
What problems were discovered during exercises?
Are exercise findings being corrected?
🚨 Useful DR / BC Measures Measure demonstrated capability rather than document existence alone
"All critical systems have DR plans."
"92% of critical services demonstrated recovery within their approved RTO during the most recent exercise cycle."
The DR Exercise "Passed"
A business service was successfully restored during an exercise.
Counts vs Rates
500 endpoints are missing critical patches.
500 of 100,000 endpoints are missing critical patches.
0.5%
Vulnerable endpoints: 500
Total endpoints: 100,000
Rate: 0.5%
Vulnerable endpoints: 200
Total endpoints: 1,000
Rate: 20%
➗ The Denominator Problem Percentages are only useful if you understand what population they represent
"98% of servers meet the secure configuration standard."
Ask:
- Are all servers included?
- Are cloud workloads included?
- Are newly discovered assets excluded?
- Are unsupported systems excluded from the denominator?
- Are high-risk systems concentrated in the remaining 2%?
Percentage
⚖️ Normalise for Meaningful Comparison Different-sized environments may require rates instead of raw totals
Business Unit A: 40 incidents across 20,000 users.
Business Unit B: 20 incidents across 500 users.
Comparing only 40 versus 20 may be misleading.
Useful normalised measures might include:
Trend Is Often More Useful Than a Snapshot
January: 40
February: 62
March: 91
April: 138
Trend
Targets, Thresholds & Tolerances
The desired performance level.
99% of critical patches deployed within 14 days.
A defined level that triggers attention or action.
Escalate if compliance falls below 95%.
The amount of variation or exposure the organisation is prepared to accept.
Zero overdue critical vulnerabilities on externally exposed payment systems.
Bad Data Produces Bad Metrics
Does the dataset include the necessary population?
Does the data represent what actually occurred?
Is it current enough for the decision?
Is the same definition used across systems and reporting periods?
Can the reported value be connected back to its source?
Does the measure actually represent the concept it claims to measure?
Data Quality
🔄 Metric Changes May Come From Collection Changes Not every movement represents a real security change
Vulnerability scanner covers: 10,000 servers.
Critical vulnerabilities: 100.
New discovery integration expands coverage to: 30,000 servers.
Critical vulnerabilities: 280.
The apparent increase does not automatically prove that security became worse.
The organisation may simply have gained:
better visibility.
📖 Define the Metric Everyone should calculate the same thing in the same way
"Critical vulnerabilities overdue"
The definition should answer questions such as:
Document the Measure
Important security measures should have enough documentation that another competent person can understand and reproduce them.
What is the measure called?
Which objective or risk does it support?
What exactly is being measured?
How is the value calculated?
Where does the underlying information originate?
How often is it calculated or reported?
Who is responsible for the measure?
How should the value be interpreted?
🤖 Automated vs Manual Collection Automation improves scale but does not remove the need for validation
Vulnerability-management data may be collected automatically.
Risk-acceptance rationale may require manual governance information.
Security Dashboards
Dashboards aggregate measures so decision-makers can understand security performance and risk.
A Useful Dashboard Shows
Critical findings: 56
Critical findings: 56
Target: < 20
Previous month: 34
Trend: Worsening
Main concentration: Internet-facing legacy systems
👥 Match Metrics to the Audience The same security process may require different views
Needs detailed system and remediation information.
Needs process effectiveness and prioritisation.
Needs business-risk context and significant exceptions.
Aggregation Can Hide Risk
Patch compliance: 98%
Patch compliance: 72%
The enterprise-wide average looks healthy while one critical environment has significant exposure.
Aggregation
📉 Averages Can Mislead Understand the distribution behind the number
Average remediation time: 12 days.
But suppose:
- 90% are fixed in 2 days;
- 10% remain open for 102 days.
The average alone hides the long-running risk.
🧩 Correlation ≠ Causation A metric moving after an action does not automatically prove why it moved
Phishing interaction decreases after awareness training.
The training may have helped, but other variables might include:
Uncertainty Matters
Security measurements are rarely perfect representations of reality.
Sources of uncertainty can include:
Dashboard says:
"100% of known servers were scanned."
That does not prove:
"100% of all servers that actually exist were scanned."
🎮 Metrics Can Change Behaviour People may optimise the number rather than the security outcome
"Close 95% of security findings within 30 days."
Teams begin:
- closing findings before fixes are validated;
- reclassifying difficult findings;
- requesting exceptions instead of fixing issues.
Closure rate can be combined with retest failure rate, reopened findings and exception volume.
Use Complementary Metrics
One measure rarely tells the whole security story.
Activity vs Outcome
Measures work performed.
Measures whether the desired security condition improved.
Security team conducted: 80 penetration tests.
96% of critical penetration-test findings were remediated and successfully retested within the agreed target.
Activity vs Outcome
⚙️ Efficiency vs Effectiveness Doing something quickly is different from doing the right thing successfully
How economically or quickly is the security process performed?
Average access-review completion time.
Does the security process achieve its intended objective?
Percentage of inappropriate privileges identified and removed.
Measurement Frequency
Different security processes require different measurement frequencies.
📏 Establish a Baseline You need a reference point before improvement can be demonstrated
Critical vulnerability remediation: 61% within target.
Critical vulnerability remediation: 91% within target.
Baseline
🔍 Metrics Point to Questions The dashboard identifies the symptom; investigation finds the cause
Patch compliance falls:
97% → 91% → 84%
The metric tells management:
something is worsening.
Investigation may discover:
Protect Security Process Data
Security metrics and their underlying data can themselves be sensitive.
A dashboard identifying:
every internet-facing system with an overdue critical vulnerability
would be extremely useful to defenders.
It could also be extremely useful to an attacker.
🛡️ Integrity of Security Metrics Decision-makers must be able to trust the numbers
If security data can be manipulated, management may make decisions using a false view of risk.
A compromised administrator suppresses vulnerability records so that compliance appears to be:
99.9%
when the real figure is:
78%.
Security Data Pipeline
Data Pipeline
Security Dashboard for an Online Bank
Management wants to understand whether security risk is improving.
Privileged access reviews completed: 97%
Overdue internet-facing critical vulnerabilities: 23
High-severity simulated attack techniques detected: 91%
Critical systems successfully restore-tested: 95%
Phishing reporting rate: 68%
Critical services meeting RTO during exercises: 89%
The Green Dashboard
Executive management receives a security dashboard showing:
Everything is coloured green.
A deeper review reveals:
- the 2% of incomplete access reviews are privileged administrators;
- restore testing has not been performed for several critical systems;
- the 3% of unpatched systems contain the organisation's highest-value services;
- training completion is 100%, but simulated phishing reporting remains very low.
Access Reviews Improve
Quarterly access-review completion increases:
82% → 91% → 99%
Management celebrates the improvement.
Security then checks another measure:
Percentage of reviews that resulted in inappropriate access being removed.
That number falls almost to zero.
Are reviewers genuinely reviewing access or simply completing the administrative task?
99.99% Backup Success
Everyone Completed Training
🎓 CISSP Scenarios Recognise the measurement principle being tested
A security team collects millions of records but cannot explain what management should do with them.
Primary problem?
The data is not connected to a decision or objective.
Management wants to know whether the vulnerability-remediation programme is meeting its target.
Which type of indicator is particularly appropriate?
KPI.
Management wants early warning that cyber exposure is increasing.
Which type?
KRI.
Ninety-eight percent of critical patches are deployed within the target period.
This primarily measures what?
Process performance.
The number of overdue critical internet-facing vulnerabilities rises sharply for three consecutive months.
This can function as what?
A key risk indicator.
Management tracks only the number of confirmed breaches that already occurred.
What type of indicator is this?
Lagging indicator.
A rapid increase in unsupported critical systems is monitored because it may signal future security problems.
Which type?
Leading indicator.
Security reports that 400 endpoints are unpatched but does not state whether the organisation has 500 or 500,000 endpoints.
What important information is missing?
The denominator / population context.
Business Unit A has more incidents than Business Unit B, but A has fifty times as many users.
What may improve comparison?
A normalised rate.
A metric shows 99% compliance.
The remaining 1% contains every internet-facing payment server.
Which issue?
Aggregation hides concentrated risk.
Access-review completion improves from 70% to 98%.
What additional measure would help show effectiveness?
Whether inappropriate access is actually identified and removed.
A manager completes every review by selecting "Approve All" without examining permissions.
Which lesson?
Process completion does not prove control effectiveness.
The organisation measures the percentage of former-user accounts disabled within four hours.
Which official 6.3 data area?
Account management.
Security tracks the number of privileged service accounts with no owner.
Which area?
Account-management data / risk indicator.
Security measures whether risk acceptances were signed by the required authority.
Which official area?
Management review and approval.
A security exception passed its expiry date but remains active.
What could this represent?
A useful governance KRI.
Every nightly backup job reports success.
Does this prove recovery is possible?
No.
An organisation periodically restores backup information and verifies that the restored data is usable.
Which official 6.3 area?
Backup verification data.
Backup jobs are 99.9% successful, but restore tests succeed only 70% of the time.
Which measure is more directly relevant to recoverability?
The restore-test success rate.
A recovery test restores a service in six hours when the approved RTO is two hours.
Was the business recovery objective met?
No.
Annual awareness training has 100% completion.
Does that prove it is effective?
No.
Security tracks training completion, knowledge checks, phishing interactions and suspicious-message reporting.
Why is this stronger?
It combines activity, understanding and behavioural measures.
A phishing simulation suddenly shows more failures, but the simulation was made substantially harder.
What should analysts avoid?
Assuming the increase automatically proves user behaviour worsened.
Every critical application has a DR plan.
What stronger evidence is needed?
Exercise and recovery-performance data.
The number of vulnerabilities increases sharply immediately after the organisation doubles scanner coverage.
What should analysts consider?
The measurement population changed.
Two teams report "critical vulnerabilities overdue" but calculate the metric using different remediation deadlines.
Primary problem?
Inconsistent metric definition.
Management sees a highly precise metric of 99.87%, but the asset inventory feeding the calculation is incomplete.
Which issue?
Data quality and measurement uncertainty.
Security reports one month's value without previous values or a target.
What context would improve interpretation?
Trend and target/threshold information.
Critical vulnerabilities increase from 20 to 40 to 70 to 110.
Why is the trend important?
It shows worsening direction, not merely current state.
A KPI falls below a predefined level that requires escalation.
Which concept?
Threshold.
A metric is collected every five minutes, but nobody uses it to make a security decision.
Is frequent collection automatically valuable?
No.
Executives receive thousands of hostnames, CVEs and technical scanner details.
What should improve?
Reporting should be tailored to the audience and decision.
Engineers receive only an enterprise-wide red/amber/green score and cannot determine which systems require fixing.
What is missing?
Operational drill-down detail.
Average remediation time is 12 days, but several critical findings are more than 180 days old.
Which issue?
The average hides the distribution and outliers.
Teams are rewarded for closing security findings quickly and begin closing them before remediation is verified.
Which problem?
The metric is driving undesirable behaviour.
Security combines closure rate with reopened findings and retest failures.
Which approach?
Complementary / balancing metrics.
Management measures how many vulnerability scans the team performed.
Which type of measure?
Activity measure.
Management measures how much high-risk exposure was actually removed after vulnerability testing.
Which type?
Outcome-oriented measure.
A dashboard metric is manually changed so management does not see a control failure.
Which security property of the measurement system failed?
Integrity.
A vulnerability dashboard lists precise weaknesses in every internet-facing system.
How should the dashboard itself be treated?
As sensitive security information.
The security manager notices that a KPI is worsening.
What should happen next?
Investigate the cause and determine whether action is required.
Management wants to know whether a new security programme actually improved performance.
What is useful before implementation?
A baseline.
Security reports different metrics to engineers, managers and executives from the same underlying dataset.
Is this necessarily wrong?
No.
Different audiences require information appropriate to different decisions.
A security measure cannot be reproduced because nobody knows its formula or data source.
Which control would help?
Document the measure definition and calculation.
Security reports a risk metric only once every year even though the underlying environment changes daily.
Which issue?
Measurement frequency may be inappropriate.
Recognise the Clue Words
Raw Events
Source observations.
DataSecurity Performance
Against objective.
KPIChanging Exposure
Risk warning.
KRIFuture Warning
Before outcome.
Leading IndicatorOutcome Already Occurred
Historical result.
Lagging IndicatorHow Many?
Absolute number.
CountOut of How Many?
Population.
DenominatorCompare Different Sizes
Rate / ratio.
NormaliseCurrent vs Previous
Direction.
TrendDesired Performance
Goal.
TargetTrigger Action
Limit crossed.
ThresholdLeaver Accounts
Identity lifecycle.
Account ManagementRisk Approval
Governance evidence.
Management ReviewBackup Job Passed
What next?
Verify RestoreTraining Completed
Does behaviour change?
EffectivenessDR Plan Exists
Has it been demonstrated?
Exercise / Recovery DataIncomplete Asset Inventory
Metric confidence problem.
Data QualityCollection Method Changed
Trend comparability?
Measurement ChangeEnterprise Average Looks Good
Critical area bad.
Aggregation RiskNumber Drives Wrong Behaviour
Optimising metric.
Metric GamingScans Performed
Work done.
ActivityExposure Reduced
Security result.
Outcome⚠️ Common CISSP Mistakes Numbers without context can create false confidence
Raw observations need context and purpose before they become useful measures.
Collect data that supports relevant security decisions.
KPI primarily communicates performance.
KRI primarily communicates risk exposure.
A precise number can still be based on incomplete or poor-quality data.
Raw totals may need a denominator to provide meaningful context.
Understand which population is included and where failures are concentrated.
Historical direction may reveal worsening risk that a single measurement does not.
A target describes desired performance.
A threshold commonly defines when attention or action is required.
Completed reviews, training or approvals may still fail to achieve their security objective.
Successful backup jobs do not prove that usable recovery can occur.
Completion measures attendance rather than behavioural outcome.
Exercise data and actual recovery performance provide stronger evidence.
Compare actual recovery performance against business requirements.
Collection scope, definitions or technology may have changed.
Verify the formula, denominator, scope and definitions before comparing values.
Aggregation can hide concentrated high-risk areas.
Outliers and long tails may be hidden by a single average value.
Investigate other variables before assuming why a metric changed.
Metrics identify conditions and trends; investigation often explains why they occurred.
Number of scans, tests or training sessions does not itself prove that risk was reduced.
Efficiency and effectiveness are different properties.
Automated data still requires validation, consistent definitions and appropriate interpretation.
Reporting should lead to appropriate risk or operational action.
Security-process information can reveal sensitive vulnerabilities and control weaknesses.
Quick Reference
| If you see... | Think... |
|---|---|
| Raw records | Data |
| Performance against objective | KPI |
| Changing risk exposure | KRI |
| Possible future warning | Leading Indicator |
| Past outcome | Lagging Indicator |
| What did we do? | Activity Measure |
| What improved? | Outcome Measure |
| How fast / economically? | Efficiency |
| Did it achieve objective? | Effectiveness |
| Absolute number | Count |
| Out of how many? | Denominator |
| Compare different population sizes | Normalise |
| Reference before improvement | Baseline |
| Desired level | Target |
| Action-triggering level | Threshold |
| Values over time | Trend |
| Former-user revocation | Account Management Data |
| Approval and review status | Management Review Data |
| Backup job successful | Verify Restore |
| Training attendance | Completion |
| Did training change behaviour? | Effectiveness |
| DR plan tested | Recovery Evidence |
| How much data can be lost? | RPO |
| How quickly must service recover? | RTO |
| Missing assets in dataset | Completeness Problem |
| Different calculation methods | Consistency Problem |
| Aggregate looks good, critical area bad | Drill Down |
| Metric drives bad behaviour | Metric Gaming |
KPI & KRI Memory Aid
6.3 Official Topics Memory Aid
Data Quality Memory Aid
6.3 Master Memory Aid
Data → Measure → Indicator → Decision
The Security Metrics Questions
Key Takeaways
CISSP 6.3 focuses on collecting technical and administrative security process data and turning that data into useful security information.
The current CISSP outline explicitly identifies account management, management review and approval, key performance and risk indicators, backup verification data, training and awareness, and Disaster Recovery / Business Continuity.
Security data is valuable when it supports a security, risk or management decision.
Raw observations can be transformed into measures, metrics and indicators that provide context about security performance and risk.
Data → Measure → Indicator → Decision.
Technical security data can come from vulnerability tools, authentication systems, backups, endpoints, networks, applications and security monitoring.
Administrative security data can include access reviews, approvals, exceptions, training, risk decisions and recovery exercises.
A useful metric should be relevant to an objective, based on reliable data, repeatable, understandable, timely and actionable.
Collecting a large volume of security data does not automatically create a useful security measurement programme.
Interesting numbers are not necessarily useful metrics.
Key Performance Indicators primarily communicate how well a process, control or programme is performing against an objective.
Key Risk Indicators primarily communicate the level or movement of risk exposure.
KPI = performance. KRI = risk.
The same security process may produce both performance and risk indicators depending on the decision being supported.
Leading indicators can provide warning about possible future conditions, while lagging indicators measure outcomes that have already occurred.
Security measures can be quantitative or qualitative.
A precise numerical value should not automatically be assumed to be more accurate than a qualitative assessment.
Account-management data can reveal how effectively identities and access are created, reviewed and removed.
Useful account measures can include deprovisioning time, dormant accounts, orphaned accounts, privileged identities and access-review completion.
Management review and approval data can demonstrate whether required governance activities occur.
Review completion does not automatically prove review effectiveness.
Combining process-completion metrics with outcome metrics can provide stronger assurance.
Backup verification data should include more than backup-job completion.
Restore testing provides stronger evidence that backup information can actually support recovery.
Backup success ≠ recovery success.
RPO relates to acceptable data loss measured in time, while RTO relates to the target time for restoring a service or capability.
A system that eventually recovers may still fail its business recovery requirement if the RTO is exceeded.
Training and awareness data should extend beyond simple attendance where the objective is to understand programme effectiveness.
Completion, knowledge, behavioural and reporting measures can provide complementary views of awareness effectiveness.
Training completed ≠ behaviour changed.
DR and BC data can include plan coverage, exercise completion, recovery performance, RTO/RPO achievement and remediation of exercise findings.
Having a DR plan does not itself prove that recovery capability exists.
Raw counts can be misleading when populations differ.
Rates and normalised measures can support more meaningful comparisons.
Always understand the denominator.
A percentage is only meaningful when the population included in the calculation is understood.
Trends frequently provide more information than individual snapshots.
A worsening sequence of values can signal increasing exposure even before an explicit threshold is crossed.
Targets describe desired performance.
Thresholds can define levels at which attention, escalation or action is required.
Security measurements depend on data quality.
Complete, accurate, timely and consistent data produces more reliable measures.
Changes in data sources, coverage, definitions or calculation methods can change reported metrics even when underlying security has not changed.
Important metrics should therefore have documented definitions, formulas, data sources, owners and reporting frequency.
Automated collection improves scalability and timeliness, but automated data still needs appropriate validation and interpretation.
Dashboards should provide context such as current value, target, trend, threshold and required action.
Different audiences need different presentations of the same underlying security data.
Engineers generally need operational detail, while senior management generally requires aggregation and business-risk context.
Aggregate for management, but preserve drill-down for risk.
Enterprise averages can conceal high-risk concentrations in critical environments.
Averages may also conceal outliers and long-running problems, so distributions, age bands, medians or percentiles may sometimes provide additional insight.
Metric changes should not automatically be interpreted as causation. Changes in the test, population, technology or collection process may also explain the result.
Security measurements contain uncertainty because inventories, observations and collection systems may be incomplete.
Precision of the number does not guarantee completeness of reality.
Metrics can influence behaviour, sometimes in undesirable ways.
Complementary measures can reduce the risk that teams optimise a single number rather than the underlying security outcome.
Activity measures show what the security programme did.
Outcome measures show what security condition changed.
Number of scans performed is activity. Exposure reduced is outcome.
Efficiency and effectiveness are different: a process can operate quickly while still failing to achieve the security objective.
Measurement frequency should match the rate of change, risk and decision need.
Baselines provide a reference against which improvement or deterioration can be evaluated.
Metrics frequently identify symptoms rather than root causes.
A worsening metric should therefore trigger appropriate investigation rather than unsupported assumptions.
Security process data and dashboards can contain sensitive information about weaknesses, privileged accounts and critical infrastructure.
Their confidentiality, integrity and access should therefore be protected.
If management cannot trust the measurement, management cannot safely trust the decision based on it.
The central CISSP principle is: collect reliable security-process data, transform it into meaningful measures, interpret those measures in context and use them to make better security and risk decisions.
📚 Sources & Further Reading Current security measurement and risk references
-
ISC2 - CISSP Certification Exam Outline
View the current CISSP Exam Outline -
NIST SP 800-55 Vol. 1 - Measurement Guide for Information Security:
Identifying and Selecting Measures
View current NIST security-measurement guidance -
NIST SP 800-55 Vol. 2 - Measurement Guide for Information Security:
Developing an Information Security Measurement Program
View current NIST measurement-program guidance -
NIST IR 8286C Rev. 1 - Staging Cybersecurity Risks for Enterprise
Risk Management and Governance Oversight
View NIST cybersecurity-risk measurement guidance -
NIST CSRC Glossary - Key Risk Indicator
View NIST KRI terminology